Nov 01 14:14:18.021169 ip-10-0-131-109 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 14:14:18.021182 ip-10-0-131-109 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 14:14:18.021190 ip-10-0-131-109 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 14:14:18.021519 ip-10-0-131-109 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 14:14:28.209511 ip-10-0-131-109 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 14:14:28.209529 ip-10-0-131-109 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d151a1b6cc4b4b19a382f841eb4cfa55 -- Nov 01 14:16:27.349497 ip-10-0-131-109 systemd[1]: Starting Kubernetes Kubelet... Nov 01 14:16:27.785045 ip-10-0-131-109 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 14:16:27.785045 ip-10-0-131-109 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 14:16:27.785045 ip-10-0-131-109 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 14:16:27.785045 ip-10-0-131-109 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 14:16:27.789288 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789194 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.789288 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789287 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.789288 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789292 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789295 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789298 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789301 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789304 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789306 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789309 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789313 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789316 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789319 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789323 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789328 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789330 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789333 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789337 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789342 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789346 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789349 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789353 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.789381 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789356 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789360 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789363 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789367 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789369 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789372 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789375 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789378 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789380 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789383 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789386 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789389 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789392 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789395 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789399 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789401 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789404 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789407 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789409 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789412 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.789856 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789414 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789417 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789420 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789423 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789426 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789428 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789431 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789434 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789436 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789439 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789442 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789445 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789447 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789451 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789454 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789457 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789460 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789462 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789465 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789467 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.790346 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789470 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789473 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789475 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789478 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789480 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789483 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789485 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789488 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789491 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789493 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789496 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789499 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789501 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789504 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789506 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789510 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789512 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789515 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789517 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.790822 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789520 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789523 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789526 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789529 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789531 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789534 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789537 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789540 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789542 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789545 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789547 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789550 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789553 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789556 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789559 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789562 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789564 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789567 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789569 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.791317 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789572 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789574 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789577 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789579 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789582 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789584 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.789587 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791135 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791144 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791147 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791159 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791162 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791166 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791169 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791171 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791175 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791178 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791180 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791183 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791185 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.791756 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791188 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791191 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791194 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791197 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791199 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791202 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791205 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791208 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791211 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791228 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791230 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791233 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791236 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791239 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791241 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791244 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791247 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791249 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791252 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791254 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.792246 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791257 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791259 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791262 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791265 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791269 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791272 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791274 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791278 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791281 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791283 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791286 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791289 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791291 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791294 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791296 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791301 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791304 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791308 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791311 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.792719 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791314 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791317 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791319 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791322 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791325 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791328 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791331 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791333 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791336 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791339 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791341 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791344 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791347 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791349 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791352 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791354 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791357 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791361 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791364 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791366 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.793170 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791370 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791374 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791379 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791382 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791386 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791388 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791392 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791394 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791397 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791400 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791402 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791405 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791408 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791411 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791413 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791416 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791419 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791421 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791424 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791426 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.793664 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791429 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791432 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791434 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791437 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791440 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791442 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791446 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791449 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791451 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791454 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791457 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791461 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791464 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791466 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791959 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791965 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791969 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791972 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791975 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.794133 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791978 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791981 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791984 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791986 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791989 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791992 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791995 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.791997 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792000 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792003 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792005 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792008 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792011 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792013 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792017 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792022 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792025 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792028 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792031 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.794594 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792034 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792036 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792039 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792042 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792045 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792048 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792051 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792054 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792057 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792059 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792062 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792065 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792067 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792071 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792074 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792077 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792080 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792082 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792085 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.795166 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792087 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792090 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792093 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792097 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792099 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792102 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792104 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792107 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792110 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792113 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792115 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792119 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792122 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792124 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792127 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792129 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792132 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792135 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792138 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792140 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.795634 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792143 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792145 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792148 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792151 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792153 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792156 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792158 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792161 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792163 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792166 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792169 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792171 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792174 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792177 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792179 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792182 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792185 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792188 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792190 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792193 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.796118 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792195 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792198 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792200 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792203 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792206 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792208 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792211 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792226 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792229 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792231 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792234 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792237 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792240 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792243 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792245 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792248 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792251 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792253 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792257 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.796623 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792260 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792263 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792265 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.792268 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793228 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793239 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793246 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793251 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793256 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793259 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793264 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793268 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793271 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793274 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793278 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793281 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793284 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793288 2583 flags.go:64] FLAG: --cgroup-root="" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793291 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793294 2583 flags.go:64] FLAG: --client-ca-file="" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793297 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793300 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793304 2583 flags.go:64] FLAG: --cluster-domain="" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793307 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 14:16:27.797064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793310 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793314 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793318 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793321 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793325 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793328 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793331 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793335 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793338 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793341 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793344 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793347 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793351 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793355 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793358 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793361 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793364 2583 flags.go:64] FLAG: --enable-server="true" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793367 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793373 2583 flags.go:64] FLAG: --event-burst="100" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793376 2583 flags.go:64] FLAG: --event-qps="50" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793379 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793382 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793385 2583 flags.go:64] FLAG: --eviction-hard="" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793390 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 14:16:27.797657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793393 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793397 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793400 2583 flags.go:64] FLAG: --eviction-soft="" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793403 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793407 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793410 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793413 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793416 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793419 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793422 2583 flags.go:64] FLAG: --feature-gates="" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793427 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793430 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793433 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793437 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793440 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793443 2583 flags.go:64] FLAG: --help="false" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793447 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793450 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793453 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793456 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793460 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793466 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793469 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 14:16:27.798282 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793472 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793475 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793478 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793481 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793484 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793487 2583 flags.go:64] FLAG: --kube-reserved="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793490 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793494 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793497 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793499 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793502 2583 flags.go:64] FLAG: --lock-file="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793505 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793508 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793511 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793517 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793520 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793524 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793527 2583 flags.go:64] FLAG: --logging-format="text" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793530 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793533 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793537 2583 flags.go:64] FLAG: --manifest-url="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793540 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793545 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793548 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793552 2583 flags.go:64] FLAG: --max-pods="110" Nov 01 14:16:27.798826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793556 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793559 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793562 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793565 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793568 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793572 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793575 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793583 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793587 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793592 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793595 2583 flags.go:64] FLAG: --pod-cidr="" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793598 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793601 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793605 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793608 2583 flags.go:64] FLAG: --port="10250" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793611 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793614 2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0396536c18092b566" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793617 2583 flags.go:64] FLAG: --qos-reserved="" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793620 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793623 2583 flags.go:64] FLAG: --register-node="true" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793626 2583 flags.go:64] FLAG: --register-with-taints="" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793631 2583 flags.go:64] FLAG: --registry-burst="10" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793634 2583 flags.go:64] FLAG: --registry-qps="5" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793637 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793640 2583 flags.go:64] FLAG: --reserved-memory="" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793644 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 14:16:27.799436 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793647 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793650 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793653 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793657 2583 flags.go:64] FLAG: --runonce="false" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793659 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793663 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793666 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793669 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793673 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793676 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793679 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793684 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793687 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793690 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793693 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793697 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793700 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793703 2583 flags.go:64] FLAG: --system-cgroups="" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793706 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793712 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793715 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793718 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793722 2583 flags.go:64] FLAG: --tls-min-version="" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793725 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793728 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 14:16:27.800046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793731 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793734 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793737 2583 flags.go:64] FLAG: --v="2" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793742 2583 flags.go:64] FLAG: --version="false" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793746 2583 flags.go:64] FLAG: --vmodule="" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793750 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.793753 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793892 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793895 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793898 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793902 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793904 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793907 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793910 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793912 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793915 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793917 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793920 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793924 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793927 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793929 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.801012 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793932 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793936 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793939 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793942 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793944 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793947 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793950 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793953 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793955 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793958 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793961 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793963 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793966 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793968 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793971 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793974 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793977 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793979 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793982 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.801955 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793984 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793987 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793990 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793993 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793996 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.793998 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794001 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794003 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794006 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794010 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794015 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794018 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794021 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794024 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794027 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794030 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794033 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794036 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794039 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.802786 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794041 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794044 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794046 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794049 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794052 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794054 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794057 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794059 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794062 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794065 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794067 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794070 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794072 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794075 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794077 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794080 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794083 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794086 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794089 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794091 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.803686 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794094 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794096 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794099 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794102 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794105 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794108 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794111 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794114 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794117 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794120 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794123 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794125 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794128 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794130 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794133 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794136 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794139 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794143 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794147 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.804568 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794150 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794153 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794156 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794159 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794162 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794164 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794167 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794170 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794173 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794175 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794178 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794181 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794184 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794187 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794189 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.794225 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 14:16:27.805159 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794265 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794269 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794272 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794275 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794278 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794282 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794285 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794288 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794290 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794293 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794296 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794298 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794301 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794304 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794307 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794309 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794312 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794315 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794318 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.805602 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794320 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794323 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794326 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794328 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794331 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794333 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794336 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794339 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794341 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794344 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794347 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794349 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794352 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794356 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794359 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794362 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794365 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794367 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794371 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.806399 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794374 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794378 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794381 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794385 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794388 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794391 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794393 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794396 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794399 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794401 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794404 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794407 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794409 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794412 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794414 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794417 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794420 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794422 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794425 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794428 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.807114 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794430 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794433 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794435 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794438 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794441 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794444 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794448 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794451 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794454 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794457 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794459 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794463 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794466 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794468 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794471 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794474 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794477 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794480 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794482 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794485 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.807709 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794487 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794491 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794494 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794496 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794499 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794502 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794504 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794507 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794511 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794515 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794518 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794521 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794524 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794527 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794529 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794532 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794535 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794538 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794543 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.808189 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794561 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794565 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794568 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794571 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794575 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794577 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794581 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794584 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.794587 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.802666 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.802686 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.802708 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.802723 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802805 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802814 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802819 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802825 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802830 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802835 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.808809 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802839 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802844 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802849 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802853 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802857 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802862 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802867 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802871 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802876 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802880 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802884 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802889 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802893 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802897 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802901 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802906 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802911 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802915 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802919 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802924 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.809323 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802928 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802932 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802937 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802943 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802948 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802953 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802957 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802962 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802966 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802971 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802976 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802981 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802985 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802989 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802994 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.802998 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803003 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803007 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803011 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.809899 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803016 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803020 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803024 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803028 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803032 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803036 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803041 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803045 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803049 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803053 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803058 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803062 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803067 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803071 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803075 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803080 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803086 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803092 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803096 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803100 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.810414 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803105 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803109 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803114 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803118 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803123 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803127 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803132 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803136 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803141 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803145 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803150 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803154 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803158 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803162 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803167 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803171 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803175 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803180 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803184 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.810900 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803188 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803192 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803197 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803201 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803205 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803209 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803229 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803234 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803238 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803243 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803248 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803254 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803260 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803265 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803269 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803275 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803282 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803287 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803292 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.811432 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803296 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803301 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803305 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.803349 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803572 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803581 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803586 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803591 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803596 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803600 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803605 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803610 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803614 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803619 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803624 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803628 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.811877 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803632 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803637 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803641 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803645 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803649 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803654 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803658 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803663 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803667 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803672 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803676 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803681 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803685 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803690 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803695 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803699 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803703 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803707 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803712 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803716 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.812528 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803721 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803725 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803729 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803734 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803738 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803743 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803747 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803754 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803760 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803766 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803771 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803777 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803782 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803786 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803790 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803794 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803799 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803803 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803808 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.813281 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803812 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803817 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803821 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803828 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803832 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803836 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803841 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803846 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803850 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803856 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803861 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803866 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803871 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803875 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803879 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803884 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803888 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803892 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803897 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.813790 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803901 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803906 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803910 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803914 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803919 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803924 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803928 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803932 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803936 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803941 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803945 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803949 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803954 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803959 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803963 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803967 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803972 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803982 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803986 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.814300 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803990 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803995 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.803999 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804003 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804008 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804012 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804017 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804021 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804025 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804029 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804033 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804038 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804042 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804046 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804051 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804055 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.814997 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804060 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.804097 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804148 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804155 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804160 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804165 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804169 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804174 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804178 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804183 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804188 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804192 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804196 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804201 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804205 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804211 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:16:27.815628 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804233 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804237 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804241 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804245 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804249 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804254 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804258 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804262 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804266 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804270 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804274 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804278 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804282 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804287 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804292 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804296 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804301 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804305 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804309 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804313 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:16:27.816302 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804317 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804322 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804326 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804330 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804335 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804340 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804344 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804348 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804354 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804361 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804366 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804372 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804377 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804382 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804386 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804390 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804394 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804399 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804403 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:16:27.816905 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804407 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804411 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804417 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804421 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804425 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804429 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804434 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804438 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804443 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804448 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804454 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804461 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804466 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804470 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804474 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804478 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804482 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804486 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804491 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:16:27.817407 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804495 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804499 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804504 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804508 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804512 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804516 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804520 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804525 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804529 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804534 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804538 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804542 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804546 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804550 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804555 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804563 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804567 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804571 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804576 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:16:27.817887 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804579 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804583 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804587 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804591 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804595 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804601 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804606 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804610 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804614 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804619 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804623 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804627 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804631 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804635 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:27.804640 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.804724 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.807562 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.810281 2583 server.go:1014] "Starting client certificate rotation" Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.810380 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 14:16:27.818347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.810447 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 14:16:27.818797 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.812572 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 14:16:27.818797 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.817116 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 14:16:27.827843 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.827825 2583 log.go:25] "Validated CRI v1 runtime API" Nov 01 14:16:27.831149 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.831134 2583 log.go:25] "Validated CRI v1 image API" Nov 01 14:16:27.832739 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.832716 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 14:16:27.837344 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.837321 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99da58a5-bd91-45f0-ada5-0ca15684927b:/dev/nvme0n1p4 dd963982-9278-49f7-8cc4-4937440009e8:/dev/nvme0n1p3] Nov 01 14:16:27.837405 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.837344 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 14:16:27.841636 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.841528 2583 manager.go:217] Machine: {Timestamp:2026-11-01 14:16:27.841045718 +0000 UTC m=+0.368418454 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101260 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24883c1eda0f9dcd42bce059f67a2e SystemUUID:ec24883c-1eda-0f9d-cd42-bce059f67a2e BootID:d151a1b6-cc4b-4b19-a382-f841eb4cfa55 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c8:d6:b5:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c8:d6:b5:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:3c:b0:1a:31:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 14:16:27.842288 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.842277 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 14:16:27.842381 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.842366 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 14:16:27.845900 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.845874 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 14:16:27.846027 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.845901 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-109.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 14:16:27.846071 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.846030 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 14:16:27.846071 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.846038 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 14:16:27.846071 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.846057 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 14:16:27.846149 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.846076 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 14:16:27.846968 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.846958 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 14:16:27.850819 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.850807 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 14:16:27.850931 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.850922 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 14:16:27.851617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.851608 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 01 14:16:27.851646 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.851628 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 14:16:27.851646 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.851639 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 14:16:27.851708 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.851648 2583 kubelet.go:405] "Adding apiserver pod source" Nov 01 14:16:27.851708 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.851657 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 14:16:27.852012 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.852000 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 14:16:27.854668 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.854648 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 14:16:27.860545 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.860532 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 14:16:27.861828 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.861816 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 14:16:27.861883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.861847 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 14:16:27.862032 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862021 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 14:16:27.862064 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862043 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 14:16:27.862091 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862078 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 14:16:27.862091 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862089 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 14:16:27.862143 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862097 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 14:16:27.862143 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862103 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 14:16:27.862706 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862698 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 14:16:27.862748 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862707 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 14:16:27.862748 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862721 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 14:16:27.862748 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862728 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 14:16:27.862748 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862742 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 14:16:27.862866 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.862751 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 14:16:27.863684 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.863674 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 14:16:27.863723 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.863686 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 14:16:27.867650 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.867638 2583 server.go:1263] "Started kubelet" Nov 01 14:16:27.867803 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.867700 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 14:16:27.867975 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.867860 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 14:16:27.868070 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.868058 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 14:16:27.868451 ip-10-0-131-109 systemd[1]: Started Kubernetes Kubelet. Nov 01 14:16:27.869397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.869363 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 14:16:27.875496 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.875473 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 01 14:16:27.877478 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:27.877458 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 14:16:27.879247 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.879234 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 14:16:27.880331 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.880316 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 14:16:27.880441 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:27.880416 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:27.880961 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.880940 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 14:16:27.880961 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.880963 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 14:16:27.881092 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.880933 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 14:16:27.881092 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881031 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 01 14:16:27.881092 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881040 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 01 14:16:27.881203 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881114 2583 factory.go:55] Registering systemd factory Nov 01 14:16:27.881203 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881130 2583 factory.go:223] Registration of the systemd container factory successfully Nov 01 14:16:27.881386 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881372 2583 factory.go:153] Registering CRI-O factory Nov 01 14:16:27.881451 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881391 2583 factory.go:223] Registration of the crio container factory successfully Nov 01 14:16:27.881451 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881443 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 14:16:27.881547 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881457 2583 factory.go:103] Registering Raw factory Nov 01 14:16:27.881547 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.881475 2583 manager.go:1199] Started watching for new ooms in manager Nov 01 14:16:27.882418 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.882387 2583 manager.go:319] Starting recovery of all containers Nov 01 14:16:27.894497 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.894482 2583 manager.go:324] Recovery completed Nov 01 14:16:27.898260 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.898197 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:27.900089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900072 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:27.900164 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900104 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:27.900164 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900118 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:27.900699 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900687 2583 cpu_manager.go:226] "Starting" policy="none" Nov 01 14:16:27.900742 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900707 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:27.900792 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900708 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 14:16:27.900792 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.900776 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 14:16:27.901425 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.901406 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:27.901499 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.901440 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:27.901499 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.901455 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:27.904518 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.904503 2583 policy_none.go:50] "Start" Nov 01 14:16:27.904574 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.904521 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 14:16:27.904574 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.904533 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 14:16:27.907443 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.907429 2583 policy_none.go:44] "Start" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.945321 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.946743 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.946765 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.946782 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:27.946818 2583 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.949893 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954163 2583 manager.go:341] "Starting Device Plugin manager" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:27.954195 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954204 2583 server.go:87] "Starting device plugin registration server" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954420 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954431 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954517 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954596 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:27.954604 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:27.955084 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 14:16:27.957785 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:27.955126 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:28.047074 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.047005 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal"] Nov 01 14:16:28.047074 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.047064 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:28.047936 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.047921 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:28.048038 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.047949 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:28.048038 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.047967 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:28.050133 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050117 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:28.050290 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050277 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.050333 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050309 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:28.050897 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050876 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:28.051002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050880 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:28.051002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050934 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:28.051002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050952 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:28.051002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.050908 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:28.051185 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.051005 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:28.053155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.053137 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.053253 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.053170 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:28.053896 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.053881 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:28.053968 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.053906 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:28.053968 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.053921 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:28.055454 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.055441 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:28.056186 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.056168 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:28.056289 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.056193 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:28.056289 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.056203 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:28.056289 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.056239 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.060347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.060327 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-109.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 14:16:28.060454 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.060359 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-109.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 14:16:28.061264 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.060313 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-109.us-west-2.compute.internal.18e3f17b275f85fa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-109.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-109.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:27.867432442 +0000 UTC m=+0.394805178,LastTimestamp:2026-11-01 14:16:27.867432442 +0000 UTC m=+0.394805178,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-109.us-west-2.compute.internal,}" Nov 01 14:16:28.081543 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.081521 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.081653 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.081547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.081653 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.081564 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbed3511ed9fe6dd871e46aab86e24ee-config\") pod \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"fbed3511ed9fe6dd871e46aab86e24ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.082758 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.082742 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.085367 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.085352 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.093181 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.093159 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 14:16:28.093360 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.093324 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 01 14:16:28.093429 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.093378 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 01 14:16:28.152931 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.152904 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 14:16:28.153041 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.152975 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d4l8v" Nov 01 14:16:28.173596 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.173573 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-109.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 14:16:28.173672 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.173529 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-109.us-west-2.compute.internal.18e3f17b2951cdbe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-109.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-109.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-109.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:27.900087742 +0000 UTC m=+0.427460482,LastTimestamp:2026-11-01 14:16:27.900087742 +0000 UTC m=+0.427460482,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-109.us-west-2.compute.internal,}" Nov 01 14:16:28.182245 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.182211 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.182297 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.182252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.182297 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.182271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbed3511ed9fe6dd871e46aab86e24ee-config\") pod \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"fbed3511ed9fe6dd871e46aab86e24ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.182297 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.182269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.182404 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.182295 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.182404 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.182328 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbed3511ed9fe6dd871e46aab86e24ee-config\") pod \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"fbed3511ed9fe6dd871e46aab86e24ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.243246 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.243208 2583 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-109.us-west-2.compute.internal\" not found" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.358873 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.358810 2583 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-109.us-west-2.compute.internal\" not found" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.386339 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.386320 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.389404 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.389389 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.444050 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.444023 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:28.444933 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.444918 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:28.445002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.444947 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:28.445002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.444960 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:28.445002 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.444985 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.501060 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.501037 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:28.501152 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.501063 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-109.us-west-2.compute.internal\": node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:28.608674 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.608647 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:28.708891 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.708858 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:28.809680 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:28.809654 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 14:16:28.809680 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.809669 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:28.810133 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.809782 2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a2ba49ef185a5444dbb27d29e6baf327-c3ad9244eeee9bec.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-131-109.us-west-2.compute.internal.18e3f17b2951cdbe\": read tcp 10.0.131.109:48464->18.116.94.36:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-131-109.us-west-2.compute.internal.18e3f17b2951cdbe default 5274 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-109.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-109.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-109.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 14:16:27 +0000 UTC,LastTimestamp:2026-11-01 14:16:28.050899559 +0000 UTC m=+0.578272301,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-109.us-west-2.compute.internal,}" Nov 01 14:16:28.909928 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:28.909841 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.011030 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.010991 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.092784 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.092754 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:16:29.093729 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.093710 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:16:29.093836 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.093741 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:16:29.093836 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.093754 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:16:29.111546 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.111519 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.158490 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.158445 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 14:11:28 +0000 UTC" deadline="2028-04-10 05:30:00.80225367 +0000 UTC" Nov 01 14:16:29.158490 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.158485 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12615h13m31.643771851s" Nov 01 14:16:29.167479 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.167440 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:16:29.212682 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.212660 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.312877 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.312839 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.337414 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.337389 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:16:29.364964 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.364947 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:16:29.413301 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.413272 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.513536 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.513475 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.614290 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.614212 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:16:29.636276 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.636256 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:16:29.658765 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:29.658732 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71522b17879685585fdaa072a2e7d0d.slice/crio-49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17 WatchSource:0}: Error finding container 49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17: Status 404 returned error can't find the container with id 49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17 Nov 01 14:16:29.666975 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.666959 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 14:16:29.669984 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:29.669961 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbed3511ed9fe6dd871e46aab86e24ee.slice/crio-55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1 WatchSource:0}: Error finding container 55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1: Status 404 returned error can't find the container with id 55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1 Nov 01 14:16:29.681495 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.681480 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:29.741526 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.741506 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 14:16:29.742592 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.742579 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:16:29.799498 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.799449 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 14:16:29.852835 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.852814 2583 apiserver.go:52] "Watching apiserver" Nov 01 14:16:29.881010 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.880986 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 14:16:29.951862 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.951804 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"fbed3511ed9fe6dd871e46aab86e24ee","Type":"ContainerStarted","Data":"55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1"} Nov 01 14:16:29.952829 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.952809 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"a71522b17879685585fdaa072a2e7d0d","Type":"ContainerStarted","Data":"49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17"} Nov 01 14:16:29.956235 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.956205 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 14:16:29.956490 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.956476 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-42stp","openshift-image-registry/node-ca-jbjh4","openshift-multus/multus-additional-cni-plugins-cmjxb","kube-system/global-pull-secret-syncer-ckfsz","openshift-ovn-kubernetes/ovnkube-node-9vsn2","kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl","openshift-multus/network-metrics-daemon-f72xc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal","openshift-multus/multus-48mxn","openshift-network-diagnostics/network-check-target-t7sh8","openshift-network-operator/iptables-alerter-528rj","kube-system/konnectivity-agent-82phh","openshift-cluster-node-tuning-operator/tuned-ch8fm"] Nov 01 14:16:29.960020 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.960002 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.962132 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.962112 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:29.964306 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.964284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.966326 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.966303 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:29.966414 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.966383 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:29.968540 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.968522 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.970996 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.970980 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.973133 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.973115 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:29.973288 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.973172 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:29.975468 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.975451 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:29.978045 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.978030 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:29.980379 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.980363 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.982800 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.982708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:29.982800 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:29.982776 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:29.985035 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.985018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:29.993357 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993340 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5373225-4d3f-451e-ab69-45fee9340973-hosts-file\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:29.993439 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-kubelet\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.993439 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993385 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-etc-kubernetes\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.993505 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.993505 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993457 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.993505 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993473 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-ovn\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.993617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993509 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.993617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993536 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-systemd\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.993617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993557 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.993617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993574 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12989f4a-35d4-4dd5-929a-add087567e74-ovn-node-metrics-cert\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.993617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993614 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-kubelet-config\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:29.993777 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993642 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-conf-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.993777 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993720 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cnibin\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.993777 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-bin\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.993878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993778 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-modprobe-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.993878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993840 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-run\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.993949 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.993884 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80966b44-0240-48ab-8528-3ebcfa9c2c12-serviceca\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:29.994583 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vgg\" (UniqueName: \"kubernetes.io/projected/f5373225-4d3f-451e-ab69-45fee9340973-kube-api-access-54vgg\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:29.994653 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994602 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysconfig\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.994653 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b241b884-12de-43ee-af05-7a92bff07b42-agent-certs\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:29.994653 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994638 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-system-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.994753 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994663 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-hostroot\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.994753 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994677 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvll\" (UniqueName: \"kubernetes.io/projected/717175a0-02eb-40d7-b696-828a3cb7fcaa-kube-api-access-clvll\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.994753 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994727 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-script-lib\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.994873 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994756 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-tmp\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.994873 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994788 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:29.994873 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994808 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-node-log\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.994959 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-device-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.994959 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.994959 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj9vt\" (UniqueName: \"kubernetes.io/projected/e8a11aea-1720-432f-a0e4-63a9acbb1efa-kube-api-access-fj9vt\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:29.995043 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-systemd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995043 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.994989 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-socket-dir-parent\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.995043 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995033 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-daemon-config\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.995132 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-kubelet\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995132 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-etc-tuned\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.995200 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995137 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.995200 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995162 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbqx8\" (UniqueName: \"kubernetes.io/projected/12989f4a-35d4-4dd5-929a-add087567e74-kube-api-access-xbqx8\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995200 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-sys\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.995333 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/700ebc96-8068-4533-9376-52fe515cbc3e-kube-api-access-pqzgz\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.995333 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.995333 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995313 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.995478 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995341 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:29.995478 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995375 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-systemd-units\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995478 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995409 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-etc-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995478 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-lib-modules\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.995478 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-host\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.995478 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995476 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-cnibin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.995684 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-k8s-cni-cncf-io\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.995684 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995538 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-env-overrides\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995684 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80966b44-0240-48ab-8528-3ebcfa9c2c12-host\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:29.995684 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995599 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-bin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.995684 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995644 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.995831 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995686 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnj8k\" (UniqueName: \"kubernetes.io/projected/80966b44-0240-48ab-8528-3ebcfa9c2c12-kube-api-access-xnj8k\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:29.995831 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995715 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995831 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995739 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-netns\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.995831 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-dbus\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:29.995831 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:29.995970 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-slash\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995970 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995873 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995970 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-config\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995970 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-kubernetes\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.995970 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-log-socket\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.995970 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995947 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.996142 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.995997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b241b884-12de-43ee-af05-7a92bff07b42-konnectivity-ca\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:29.996142 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5373225-4d3f-451e-ab69-45fee9340973-tmp-dir\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:29.996142 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-multus-certs\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.996142 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996093 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-var-lib-kubelet\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:29.996274 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996140 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-var-lib-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.996274 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.996274 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996237 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-multus\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.996274 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.996381 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjp2v\" (UniqueName: \"kubernetes.io/projected/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-kube-api-access-wjp2v\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:29.996381 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-netns\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.996381 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996339 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.996381 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996355 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-netd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:29.996381 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4cj\" (UniqueName: \"kubernetes.io/projected/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kube-api-access-mx4cj\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:29.996535 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996409 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-cni-binary-copy\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:29.996535 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:29.996437 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-conf\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.012622 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.012601 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 14:16:30.012707 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.012656 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lp9km\"" Nov 01 14:16:30.012823 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.012809 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.012856 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.012810 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.013835 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.013819 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 14:16:30.013835 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.013823 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9p5x8\"" Nov 01 14:16:30.013934 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.013833 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 14:16:30.015928 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.015913 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 14:16:30.016165 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.016153 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24rb9\"" Nov 01 14:16:30.016416 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.016404 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 14:16:30.016458 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.016438 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 14:16:30.016458 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.016442 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.016515 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.016466 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.020396 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020379 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 14:16:30.020477 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020464 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 14:16:30.020531 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020483 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.020577 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020545 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fnkb\"" Nov 01 14:16:30.020673 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020659 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 14:16:30.020705 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020671 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.020757 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.020746 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 14:16:30.058696 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.058642 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.058777 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.058748 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.058777 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.058754 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 14:16:30.059020 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.059008 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.059071 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.059031 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj7jb\"" Nov 01 14:16:30.062826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062805 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smpqh\"" Nov 01 14:16:30.062826 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062808 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.062984 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062812 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 14:16:30.062984 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062850 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.062984 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062817 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pw86\"" Nov 01 14:16:30.062984 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062919 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 14:16:30.062984 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062948 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 14:16:30.062984 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062968 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.063196 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.062986 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 14:16:30.063196 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.063067 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v4966\"" Nov 01 14:16:30.063196 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.063102 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46fn\"" Nov 01 14:16:30.063196 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.063168 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 14:16:30.063393 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.063231 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpvwm" Nov 01 14:16:30.081575 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.081556 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 14:16:30.097394 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-var-lib-kubelet\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.097459 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097402 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsqpb\" (UniqueName: \"kubernetes.io/projected/dabe07b7-a91c-42ef-aa55-2d2783da8093-kube-api-access-gsqpb\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.097459 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-var-lib-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.097459 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097459 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-multus\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-multus\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097488 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-var-lib-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097490 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-var-lib-kubelet\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097506 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097543 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjp2v\" (UniqueName: \"kubernetes.io/projected/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-kube-api-access-wjp2v\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097579 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-netns\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.097620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097613 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-netd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4cj\" (UniqueName: \"kubernetes.io/projected/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kube-api-access-mx4cj\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097633 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-netns\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097647 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097652 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-cni-binary-copy\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097667 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-netd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097687 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-conf\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5373225-4d3f-451e-ab69-45fee9340973-hosts-file\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097741 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-kubelet\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097799 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-kubelet\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097802 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5373225-4d3f-451e-ab69-45fee9340973-hosts-file\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097833 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-conf\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-etc-kubernetes\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097912 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-etc-kubernetes\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-ovn\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098051 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097942 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097957 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-systemd\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097978 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-ovn\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.097989 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098027 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12989f4a-35d4-4dd5-929a-add087567e74-ovn-node-metrics-cert\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098051 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-systemd\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098054 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-kubelet-config\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098060 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098031 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098085 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-kubelet-config\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098094 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-conf-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098120 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098133 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-conf-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098142 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-cni-binary-copy\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098172 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cnibin\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098191 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cnibin\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.098733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-bin\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098244 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-bin\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098253 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-modprobe-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098268 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-run\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098285 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dabe07b7-a91c-42ef-aa55-2d2783da8093-iptables-alerter-script\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098324 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098338 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80966b44-0240-48ab-8528-3ebcfa9c2c12-serviceca\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098361 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-modprobe-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vgg\" (UniqueName: \"kubernetes.io/projected/f5373225-4d3f-451e-ab69-45fee9340973-kube-api-access-54vgg\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysconfig\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098344 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-run\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b241b884-12de-43ee-af05-7a92bff07b42-agent-certs\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098437 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysconfig\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098446 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-system-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098468 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-hostroot\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvll\" (UniqueName: \"kubernetes.io/projected/717175a0-02eb-40d7-b696-828a3cb7fcaa-kube-api-access-clvll\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098507 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-system-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-script-lib\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.099555 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098545 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-tmp\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098587 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-hostroot\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098726 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dabe07b7-a91c-42ef-aa55-2d2783da8093-host-slash\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.098762 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098795 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80966b44-0240-48ab-8528-3ebcfa9c2c12-serviceca\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098805 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-node-log\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-node-log\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.098833 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:16:30.59881382 +0000 UTC m=+3.126186562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098876 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-device-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9vt\" (UniqueName: \"kubernetes.io/projected/e8a11aea-1720-432f-a0e4-63a9acbb1efa-kube-api-access-fj9vt\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098942 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-device-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-systemd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098975 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-systemd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.098984 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-socket-dir-parent\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099024 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.100395 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-socket-dir-parent\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099039 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-script-lib\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099052 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-daemon-config\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-kubelet\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-etc-tuned\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099237 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqx8\" (UniqueName: \"kubernetes.io/projected/12989f4a-35d4-4dd5-929a-add087567e74-kube-api-access-xbqx8\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099247 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-kubelet\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-sys\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/700ebc96-8068-4533-9376-52fe515cbc3e-kube-api-access-pqzgz\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099309 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099333 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099361 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-systemd-units\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-etc-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-lib-modules\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-host\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.100930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099487 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-daemon-config\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-cnibin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099532 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-k8s-cni-cncf-io\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-env-overrides\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80966b44-0240-48ab-8528-3ebcfa9c2c12-host\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099602 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-sys\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-bin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099651 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-k8s-cni-cncf-io\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnj8k\" (UniqueName: \"kubernetes.io/projected/80966b44-0240-48ab-8528-3ebcfa9c2c12-kube-api-access-xnj8k\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099708 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099722 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099755 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-etc-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099764 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-host\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099787 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-cnibin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099803 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-systemd-units\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101397 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099815 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-netns\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-dbus\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099879 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-slash\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099899 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80966b44-0240-48ab-8528-3ebcfa9c2c12-host\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099934 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-netns\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-config\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099949 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-dbus\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099560 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099966 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-kubernetes\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099984 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-bin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-log-socket\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.099986 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-slash\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100051 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-log-socket\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.101878 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.100061 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.100118 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:30.600102276 +0000 UTC m=+3.127475003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b241b884-12de-43ee-af05-7a92bff07b42-konnectivity-ca\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5373225-4d3f-451e-ab69-45fee9340973-tmp-dir\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-multus-certs\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100305 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-multus-certs\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100344 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100436 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-env-overrides\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100485 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-kubernetes\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100528 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-config\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5373225-4d3f-451e-ab69-45fee9340973-tmp-dir\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100842 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b241b884-12de-43ee-af05-7a92bff07b42-konnectivity-ca\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.100958 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-lib-modules\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.101959 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-tmp\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.102008 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-etc-tuned\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.102419 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.102148 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12989f4a-35d4-4dd5-929a-add087567e74-ovn-node-metrics-cert\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.102866 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.102274 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b241b884-12de-43ee-af05-7a92bff07b42-agent-certs\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:30.153911 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.153887 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:16:30.153911 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.153910 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:16:30.153911 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.153920 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:30.154167 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.153992 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:16:30.653974491 +0000 UTC m=+3.181347234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:30.155700 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.155626 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjp2v\" (UniqueName: \"kubernetes.io/projected/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-kube-api-access-wjp2v\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.155929 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.155900 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vgg\" (UniqueName: \"kubernetes.io/projected/f5373225-4d3f-451e-ab69-45fee9340973-kube-api-access-54vgg\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.156322 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.156292 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4cj\" (UniqueName: \"kubernetes.io/projected/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kube-api-access-mx4cj\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.156417 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.156382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9vt\" (UniqueName: \"kubernetes.io/projected/e8a11aea-1720-432f-a0e4-63a9acbb1efa-kube-api-access-fj9vt\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:30.156551 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.156533 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvll\" (UniqueName: \"kubernetes.io/projected/717175a0-02eb-40d7-b696-828a3cb7fcaa-kube-api-access-clvll\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.156840 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.156825 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnj8k\" (UniqueName: \"kubernetes.io/projected/80966b44-0240-48ab-8528-3ebcfa9c2c12-kube-api-access-xnj8k\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.157042 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.157026 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/700ebc96-8068-4533-9376-52fe515cbc3e-kube-api-access-pqzgz\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.157078 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.157041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqx8\" (UniqueName: \"kubernetes.io/projected/12989f4a-35d4-4dd5-929a-add087567e74-kube-api-access-xbqx8\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.200739 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.200716 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dabe07b7-a91c-42ef-aa55-2d2783da8093-iptables-alerter-script\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.200822 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.200764 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dabe07b7-a91c-42ef-aa55-2d2783da8093-host-slash\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.200822 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.200806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqpb\" (UniqueName: \"kubernetes.io/projected/dabe07b7-a91c-42ef-aa55-2d2783da8093-kube-api-access-gsqpb\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.200904 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.200880 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dabe07b7-a91c-42ef-aa55-2d2783da8093-host-slash\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.201201 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.201184 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dabe07b7-a91c-42ef-aa55-2d2783da8093-iptables-alerter-script\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.256784 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.256764 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqpb\" (UniqueName: \"kubernetes.io/projected/dabe07b7-a91c-42ef-aa55-2d2783da8093-kube-api-access-gsqpb\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.274852 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.274833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:16:30.276998 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.276981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:30.285674 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.285652 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb9899_1884_4d74_8951_1a1c4a56b9e9.slice/crio-8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998 WatchSource:0}: Error finding container 8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998: Status 404 returned error can't find the container with id 8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998 Nov 01 14:16:30.288237 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.288200 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb241b884_12de_43ee_af05_7a92bff07b42.slice/crio-885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6 WatchSource:0}: Error finding container 885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6: Status 404 returned error can't find the container with id 885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6 Nov 01 14:16:30.300315 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.300298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:16:30.303060 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.303044 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:16:30.305123 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.305107 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-42stp" Nov 01 14:16:30.311278 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.311260 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:16:30.311359 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.311262 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod700ebc96_8068_4533_9376_52fe515cbc3e.slice/crio-271a1bcf721b1d8863e3e37af0d6d18febe123c17a000a549e6ff4116356c93b WatchSource:0}: Error finding container 271a1bcf721b1d8863e3e37af0d6d18febe123c17a000a549e6ff4116356c93b: Status 404 returned error can't find the container with id 271a1bcf721b1d8863e3e37af0d6d18febe123c17a000a549e6ff4116356c93b Nov 01 14:16:30.317409 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.317381 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25b5ead_9a4d_489b_a628_a9fa2eca2b2d.slice/crio-13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368 WatchSource:0}: Error finding container 13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368: Status 404 returned error can't find the container with id 13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368 Nov 01 14:16:30.317492 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.317457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-48mxn" Nov 01 14:16:30.318390 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.318369 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5373225_4d3f_451e_ab69_45fee9340973.slice/crio-1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7 WatchSource:0}: Error finding container 1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7: Status 404 returned error can't find the container with id 1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7 Nov 01 14:16:30.319674 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.319601 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:16:30.321812 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.321790 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:30.321980 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.321965 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80966b44_0240_48ab_8528_3ebcfa9c2c12.slice/crio-93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda WatchSource:0}: Error finding container 93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda: Status 404 returned error can't find the container with id 93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda Nov 01 14:16:30.327338 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.327316 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717175a0_02eb_40d7_b696_828a3cb7fcaa.slice/crio-ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327 WatchSource:0}: Error finding container ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327: Status 404 returned error can't find the container with id ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327 Nov 01 14:16:30.329908 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.329888 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabe07b7_a91c_42ef_aa55_2d2783da8093.slice/crio-e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721 WatchSource:0}: Error finding container e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721: Status 404 returned error can't find the container with id e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721 Nov 01 14:16:30.332642 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:16:30.332622 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12989f4a_35d4_4dd5_929a_add087567e74.slice/crio-25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883 WatchSource:0}: Error finding container 25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883: Status 404 returned error can't find the container with id 25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883 Nov 01 14:16:30.603983 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.603901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:30.604145 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.603952 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:30.604145 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.604073 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:30.604145 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.604141 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:16:31.604127227 +0000 UTC m=+4.131499966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:30.604300 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.604082 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:30.604300 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.604179 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:31.604172873 +0000 UTC m=+4.131545595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:30.705261 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.705210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:30.705425 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.705327 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:16:30.705425 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.705344 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:16:30.705425 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.705354 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:30.705425 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:30.705405 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:16:31.705387709 +0000 UTC m=+4.232760449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:30.955858 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.955783 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883"} Nov 01 14:16:30.957211 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.957179 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-48mxn" event={"ID":"717175a0-02eb-40d7-b696-828a3cb7fcaa","Type":"ContainerStarted","Data":"ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327"} Nov 01 14:16:30.958615 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.958589 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerStarted","Data":"13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368"} Nov 01 14:16:30.959783 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.959753 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-528rj" event={"ID":"dabe07b7-a91c-42ef-aa55-2d2783da8093","Type":"ContainerStarted","Data":"e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721"} Nov 01 14:16:30.960886 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.960861 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbjh4" event={"ID":"80966b44-0240-48ab-8528-3ebcfa9c2c12","Type":"ContainerStarted","Data":"93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda"} Nov 01 14:16:30.962229 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.962181 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42stp" event={"ID":"f5373225-4d3f-451e-ab69-45fee9340973","Type":"ContainerStarted","Data":"1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7"} Nov 01 14:16:30.963369 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.963345 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" event={"ID":"700ebc96-8068-4533-9376-52fe515cbc3e","Type":"ContainerStarted","Data":"271a1bcf721b1d8863e3e37af0d6d18febe123c17a000a549e6ff4116356c93b"} Nov 01 14:16:30.964887 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.964868 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82phh" event={"ID":"b241b884-12de-43ee-af05-7a92bff07b42","Type":"ContainerStarted","Data":"885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6"} Nov 01 14:16:30.966320 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:30.966290 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" event={"ID":"35cb9899-1884-4d74-8951-1a1c4a56b9e9","Type":"ContainerStarted","Data":"8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998"} Nov 01 14:16:31.066611 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.066572 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:29 +0000 UTC" deadline="2028-05-04 13:17:58.705790271 +0000 UTC" Nov 01 14:16:31.066611 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.066600 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13199h1m27.63919325s" Nov 01 14:16:31.612473 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.612417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:31.612694 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.612559 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:31.612835 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.612706 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:31.612835 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.612770 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:33.612750248 +0000 UTC m=+6.140122987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:31.612956 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.612846 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:31.612956 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.612883 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:16:33.612870997 +0000 UTC m=+6.140243738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:31.713612 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.713578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:31.713786 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.713764 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:16:31.713849 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.713788 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:16:31.713849 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.713800 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:31.713944 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.713863 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:16:33.713843913 +0000 UTC m=+6.241216651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:31.950375 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.948478 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:31.950375 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.948622 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:31.950375 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.948707 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:31.950375 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.948787 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:31.957582 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:31.957541 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:31.957991 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:31.957752 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:32.067733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:32.067666 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:29 +0000 UTC" deadline="2028-03-29 17:56:41.107346437 +0000 UTC" Nov 01 14:16:32.067733 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:32.067731 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12339h40m9.039619202s" Nov 01 14:16:33.631117 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.630884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:33.631605 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.631147 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:33.631605 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.631260 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:37.631239358 +0000 UTC m=+10.158612096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:33.631605 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.631264 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:33.631605 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.631164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:33.631605 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.631308 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:16:37.631296776 +0000 UTC m=+10.158669525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:33.732632 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.732284 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:33.732632 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.732489 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:16:33.732632 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.732513 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:16:33.732632 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.732526 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:33.732632 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.732584 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:16:37.732565843 +0000 UTC m=+10.259938580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:33.960272 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.949351 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:33.960272 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.949447 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:33.960272 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.949501 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:33.960272 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.949545 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:33.960272 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.949653 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:33.960272 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:33.949711 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:33.999905 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:33.999869 2583 generic.go:356] "Generic (PLEG): container finished" podID="a71522b17879685585fdaa072a2e7d0d" containerID="bd3bb9834634f3d4c6127f08f9ce422e30668232b9a3238a573ef90061811ca4" exitCode=0 Nov 01 14:16:34.000203 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:34.000177 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"a71522b17879685585fdaa072a2e7d0d","Type":"ContainerDied","Data":"bd3bb9834634f3d4c6127f08f9ce422e30668232b9a3238a573ef90061811ca4"} Nov 01 14:16:34.012021 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:34.011988 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerStarted","Data":"924a4d08bca526bd8f0e1efd5f8ed40293e0239c0b9a86871318e53d95a55631"} Nov 01 14:16:35.018920 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.018885 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"a71522b17879685585fdaa072a2e7d0d","Type":"ContainerStarted","Data":"5aef96539c7b00e8ee67f1736b2ce58bd5acb585f0dc8152c999197f9ad77aed"} Nov 01 14:16:35.028448 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.028417 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="924a4d08bca526bd8f0e1efd5f8ed40293e0239c0b9a86871318e53d95a55631" exitCode=0 Nov 01 14:16:35.028552 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.028503 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerDied","Data":"924a4d08bca526bd8f0e1efd5f8ed40293e0239c0b9a86871318e53d95a55631"} Nov 01 14:16:35.040896 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.039347 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"fbed3511ed9fe6dd871e46aab86e24ee","Type":"ContainerStarted","Data":"42b8554dffb0abc8b66679ae253f9f9019c98e7e6a04019e2fa616d4446442e6"} Nov 01 14:16:35.132009 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.131427 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" podStartSLOduration=6.131405731 podStartE2EDuration="6.131405731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:35.130727964 +0000 UTC m=+7.658100711" watchObservedRunningTime="2026-11-01 14:16:35.131405731 +0000 UTC m=+7.658778477" Nov 01 14:16:35.948815 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.948370 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:35.948815 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.948441 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:35.948815 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:35.948446 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:35.948815 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:35.948544 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:35.948815 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:35.948635 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:35.948815 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:35.948722 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:36.046853 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:36.046369 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-82phh" event={"ID":"b241b884-12de-43ee-af05-7a92bff07b42","Type":"ContainerStarted","Data":"f4b09ab0a5afbf2f00dcd40594cbf8fcb1e95bc2dbeb11a501ff29da91b6f135"} Nov 01 14:16:36.159964 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:36.159470 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" podStartSLOduration=7.159200971 podStartE2EDuration="7.159200971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:35.359421516 +0000 UTC m=+7.886794264" watchObservedRunningTime="2026-11-01 14:16:36.159200971 +0000 UTC m=+8.686573717" Nov 01 14:16:37.052168 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.052083 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbjh4" event={"ID":"80966b44-0240-48ab-8528-3ebcfa9c2c12","Type":"ContainerStarted","Data":"d71e933613afd9cc0f48d9f986409755648d9fd603194095587e0333148f9e1a"} Nov 01 14:16:37.055908 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.055858 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" event={"ID":"35cb9899-1884-4d74-8951-1a1c4a56b9e9","Type":"ContainerStarted","Data":"1ad8b984b44a92b5f874f48c6ba797613d152e55647636c0d0550a2ee82f600b"} Nov 01 14:16:37.164957 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.164895 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-82phh" podStartSLOduration=4.466429785 podStartE2EDuration="9.164874239s" totalImagesPullingTime="4.698444454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:36.159941391 +0000 UTC m=+8.687314131" watchObservedRunningTime="2026-11-01 14:16:37.164874239 +0000 UTC m=+9.692246985" Nov 01 14:16:37.675789 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.675749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:37.675983 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.675817 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:37.676045 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.675985 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:37.676093 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.676083 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.676063484 +0000 UTC m=+18.203436222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:37.676190 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.676176 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:37.676269 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.676248 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.676234849 +0000 UTC m=+18.203607586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:37.776452 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.776420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:37.776614 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.776580 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:16:37.776614 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.776599 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:16:37.776614 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.776611 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:37.776774 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.776659 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:16:45.776647271 +0000 UTC m=+18.304019994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:37.950390 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.950120 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:37.950390 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.950268 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:37.950390 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.950296 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:37.950390 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.950368 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:37.950390 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:37.950393 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:37.951572 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:37.951466 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:39.045590 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.045561 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 14:16:39.062930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.062900 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-42stp" event={"ID":"f5373225-4d3f-451e-ab69-45fee9340973","Type":"ContainerStarted","Data":"52fdb30aac41dfe2e99ae88f9bf0f78063f0982fc888e01bd06f0c5f5a94a124"} Nov 01 14:16:39.072233 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.072193 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" event={"ID":"35cb9899-1884-4d74-8951-1a1c4a56b9e9","Type":"ContainerStarted","Data":"5ff8c941f6cd7f75d6c7d9b41886c0939866286ac508eae7ed7220bb592f1eac"} Nov 01 14:16:39.077241 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.077201 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-48mxn" event={"ID":"717175a0-02eb-40d7-b696-828a3cb7fcaa","Type":"ContainerStarted","Data":"04a6cf7a72075ed0723873d021dbd573fd7c6f46cac423e1ade57e22ca36cdbe"} Nov 01 14:16:39.174863 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.174819 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-42stp" podStartSLOduration=2.761057653 podStartE2EDuration="11.174803414s" totalImagesPullingTime="8.413745761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:39.174591968 +0000 UTC m=+11.701964726" watchObservedRunningTime="2026-11-01 14:16:39.174803414 +0000 UTC m=+11.702176160" Nov 01 14:16:39.175045 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.174889 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jbjh4" podStartSLOduration=4.968852112 podStartE2EDuration="11.174885194s" totalImagesPullingTime="6.206033082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:37.165342984 +0000 UTC m=+9.692715730" watchObservedRunningTime="2026-11-01 14:16:39.174885194 +0000 UTC m=+11.702257939" Nov 01 14:16:39.288267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.288177 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-48mxn" podStartSLOduration=3.228348805 podStartE2EDuration="11.288163919s" totalImagesPullingTime="8.059815114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:39.287958546 +0000 UTC m=+11.815331283" watchObservedRunningTime="2026-11-01 14:16:39.288163919 +0000 UTC m=+11.815536663" Nov 01 14:16:39.593760 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.593588 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:39.594353 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.594330 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:39.947580 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.947505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:39.947735 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:39.947638 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:39.947735 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.947512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:39.947847 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:39.947748 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:39.947847 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.947765 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:39.947847 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:39.947830 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:39.969498 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.969348 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T14:16:39.045583956Z","UUID":"46d76f89-58d0-47f7-8e23-097f9b1302f2","Handler":null,"Name":"","Endpoint":""} Nov 01 14:16:39.970939 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.970913 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 14:16:39.971055 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:39.970948 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 14:16:40.082096 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:40.082067 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="a4d1e092540830825b1016ed4ba0f103439773957506a1124126f3ff2e5ba0a2" exitCode=0 Nov 01 14:16:40.082535 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:40.082353 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerDied","Data":"a4d1e092540830825b1016ed4ba0f103439773957506a1124126f3ff2e5ba0a2"} Nov 01 14:16:41.086502 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.086416 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-528rj" event={"ID":"dabe07b7-a91c-42ef-aa55-2d2783da8093","Type":"ContainerStarted","Data":"14017ba2ed4ac39ca2f525d9914492a6c15bb154a32210c2670db32419fb1fdc"} Nov 01 14:16:41.088929 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.088899 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" event={"ID":"35cb9899-1884-4d74-8951-1a1c4a56b9e9","Type":"ContainerStarted","Data":"124a4e2d74fafec658cde74e6ea39615ac9644f83622f248341a41f38359b4b0"} Nov 01 14:16:41.197618 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.197561 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-528rj" podStartSLOduration=4.460420657 podStartE2EDuration="13.197544781s" totalImagesPullingTime="8.737124124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:41.196915262 +0000 UTC m=+13.724288008" watchObservedRunningTime="2026-11-01 14:16:41.197544781 +0000 UTC m=+13.724917526" Nov 01 14:16:41.314256 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.314172 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" podStartSLOduration=3.267136483 podStartE2EDuration="13.314149652s" totalImagesPullingTime="10.047013169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:41.313231571 +0000 UTC m=+13.840604316" watchObservedRunningTime="2026-11-01 14:16:41.314149652 +0000 UTC m=+13.841522398" Nov 01 14:16:41.949630 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.947777 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:41.949630 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.947778 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:41.949630 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:41.947914 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:41.949630 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:41.947921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:41.949630 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:41.948004 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:41.949630 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:41.948085 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:42.092906 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:42.092847 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" event={"ID":"700ebc96-8068-4533-9376-52fe515cbc3e","Type":"ContainerStarted","Data":"a3e303d9aab204fb92541b28c3f00c36574a36a68d63790bf6cd91aa16186bd1"} Nov 01 14:16:42.095146 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:42.095117 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="2ad02c6faa02eb0b9d0853d25b9d01646e20c5aa51c059877c5938cc5e35aa1b" exitCode=0 Nov 01 14:16:42.095310 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:42.095201 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerDied","Data":"2ad02c6faa02eb0b9d0853d25b9d01646e20c5aa51c059877c5938cc5e35aa1b"} Nov 01 14:16:42.207426 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:42.207327 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" podStartSLOduration=3.291067944 podStartE2EDuration="14.207307496s" totalImagesPullingTime="10.916239552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:42.20509858 +0000 UTC m=+14.732471324" watchObservedRunningTime="2026-11-01 14:16:42.207307496 +0000 UTC m=+14.734680243" Nov 01 14:16:43.948324 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:43.948065 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:43.948805 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:43.948088 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:43.948805 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:43.948114 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:43.948805 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:43.948439 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:43.948805 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:43.948556 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:43.948805 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:43.948630 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:44.102449 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:44.102333 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="e47432d018708189c273a8109d3085793b23cc663d971c8ce21359cc1e9bcf6a" exitCode=0 Nov 01 14:16:44.102449 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:44.102389 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerDied","Data":"e47432d018708189c273a8109d3085793b23cc663d971c8ce21359cc1e9bcf6a"} Nov 01 14:16:45.737037 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.737006 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:45.737413 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.737087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:45.737413 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.737205 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:45.737413 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.737275 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.737256804 +0000 UTC m=+34.264629541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:16:45.737413 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.737333 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:45.737413 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.737368 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.737357479 +0000 UTC m=+34.264730204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:16:45.837522 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.837350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:45.837644 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.837617 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:16:45.837703 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.837646 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:16:45.837703 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.837660 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:45.837770 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.837721 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:17:01.83770216 +0000 UTC m=+34.365074904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:16:45.948072 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.948043 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:45.948180 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.948069 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:45.948273 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.948237 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:45.948354 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.948289 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:45.948387 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.948373 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:45.948466 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:45.948449 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:45.972048 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:45.972012 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:46.111558 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.111522 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"054b1c5c4e263438dc68495c6e07c1d8228fc99096cac8deccbc279dbf351f82"} Nov 01 14:16:46.111718 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.111620 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"1ac1916e897e8474a2ca38e2135ebc3fd95b310de7922bbf6d289afb23cd3e19"} Nov 01 14:16:46.111718 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.111674 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"a708511a3605cf95d91eb857954168522132b4a4e66a2d6cc4cd3580f6634558"} Nov 01 14:16:46.111817 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.111725 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"c181105547b92abc7cc28595869cbd6f6bd4ddb808f4359fc2ed45444abcaa84"} Nov 01 14:16:46.111817 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.111773 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"521059994f72c29764551fba7eea68a9392e1a568f86948be4f45a876f6f9a53"} Nov 01 14:16:46.111926 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.111822 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"cb44d74b32886e15faae411a70a051a643fbe66894072e2a45e24cef6c1593fa"} Nov 01 14:16:46.172311 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:46.172239 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:47.116449 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:47.116415 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="d114e7466a84788d53b975085bddf939578542b161fd3795eae15078777c668d" exitCode=0 Nov 01 14:16:47.116928 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:47.116455 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerDied","Data":"d114e7466a84788d53b975085bddf939578542b161fd3795eae15078777c668d"} Nov 01 14:16:47.948471 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:47.948434 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:47.948646 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:47.948523 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:47.948646 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:47.948594 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:47.948784 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:47.948673 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:47.948784 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:47.948687 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:47.948886 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:47.948782 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:48.125799 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:48.125770 2583 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="dd7a29076bbca7d8c7cc135dd652913784404cc7610bb527ae917936b82f0d22" exitCode=0 Nov 01 14:16:48.126209 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:48.125808 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerDied","Data":"dd7a29076bbca7d8c7cc135dd652913784404cc7610bb527ae917936b82f0d22"} Nov 01 14:16:49.130811 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.130773 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"7964642af486cd50f81bc020bbcd0e9d5b9af024e538851b7ac555eadddb4f0d"} Nov 01 14:16:49.133740 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.133713 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" event={"ID":"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d","Type":"ContainerStarted","Data":"a21a9a2b6c5f879877ce87f800ca5b196ba609b698da72711c1e5c6e6f3b1fc1"} Nov 01 14:16:49.352305 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.352251 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cmjxb" podStartSLOduration=8.238713056 podStartE2EDuration="21.35223828s" totalImagesPullingTime="13.113525224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:49.302060201 +0000 UTC m=+21.829432946" watchObservedRunningTime="2026-11-01 14:16:49.35223828 +0000 UTC m=+21.879611025" Nov 01 14:16:49.480016 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.479916 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:49.641721 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.641691 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:49.947613 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.947560 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:49.947814 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.947635 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:49.947814 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:49.947640 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:49.947814 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:49.947738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:49.947971 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:49.947829 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:49.947971 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:49.947906 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:50.489101 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:50.488985 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:50.649028 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:50.648992 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:50.812276 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:50.812248 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:50.967340 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:50.967308 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:51.127289 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.127206 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:51.142024 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.141993 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" event={"ID":"12989f4a-35d4-4dd5-929a-add087567e74","Type":"ContainerStarted","Data":"fe1662d98f14ce2907a1a06deaf3daede16e6140c25395423a8eee51f2ef8ac3"} Nov 01 14:16:51.142333 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.142318 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:51.157547 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.157520 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:51.411590 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.411494 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" podStartSLOduration=8.078474521 podStartE2EDuration="23.411477472s" totalImagesPullingTime="15.333002951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:16:51.397466401 +0000 UTC m=+23.924839148" watchObservedRunningTime="2026-11-01 14:16:51.411477472 +0000 UTC m=+23.938850218" Nov 01 14:16:51.547158 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.547124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:51.947485 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.947458 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:51.947657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.947504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:51.947657 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.947614 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:51.947657 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:51.947626 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:51.947792 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:51.947730 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:51.947841 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:51.947814 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:51.967887 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:51.967862 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:52.144386 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.144356 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 14:16:52.144822 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.144804 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:52.159685 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.159663 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:52.165288 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.165261 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:52.327480 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.327452 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:52.570290 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.570266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:52.984620 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.984549 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7sh8"] Nov 01 14:16:52.984805 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.984656 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:52.984805 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:52.984751 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:52.985237 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.985196 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ckfsz"] Nov 01 14:16:52.985432 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.985414 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:52.985539 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:52.985514 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:52.987862 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.987841 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f72xc"] Nov 01 14:16:52.987961 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:52.987934 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:52.988046 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:52.988026 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:53.145982 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:53.145955 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 14:16:53.149369 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:53.149328 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:53.149737 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:53.149722 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 14:16:53.150233 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:53.150196 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:16:53.766692 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:53.766667 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-42stp_f5373225-4d3f-451e-ab69-45fee9340973/dns-node-resolver/0.log" Nov 01 14:16:54.148449 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:54.148374 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 14:16:54.947571 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:54.947544 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:54.947571 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:54.947580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:54.948037 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:54.947664 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:54.948037 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:54.947658 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:54.948037 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:54.947777 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:54.948037 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:54.947895 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:55.167251 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:55.167227 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbjh4_80966b44-0240-48ab-8528-3ebcfa9c2c12/node-ca/0.log" Nov 01 14:16:55.417008 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:55.416980 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:55.575373 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:55.575337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:56.011286 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.011258 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:16:56.011696 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.011480 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 14:16:56.022320 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.022287 2583 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" podUID="12989f4a-35d4-4dd5-929a-add087567e74" containerName="ovnkube-controller" probeResult="failure" output="" Nov 01 14:16:56.030754 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.030725 2583 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" podUID="12989f4a-35d4-4dd5-929a-add087567e74" containerName="ovnkube-controller" probeResult="failure" output="" Nov 01 14:16:56.053127 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.053107 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:56.256804 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.256774 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:56.516022 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.515992 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:56.947361 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.947320 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:56.947523 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.947342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:56.947523 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:56.947452 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:56.947523 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:56.947500 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:56.947617 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:56.947583 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:56.947667 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:56.947651 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:57.047573 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:57.047543 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:57.306504 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:57.306464 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:57.749161 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:57.749069 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:57.913654 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:57.913617 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:58.106511 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.106298 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:58.509417 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.509322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:58.782876 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.782806 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:58.947185 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.947156 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:16:58.947374 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.947261 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:16:58.947374 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:58.947267 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:16:58.947374 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.947342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:16:58.947495 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:58.947339 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:16:58.947495 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:16:58.947417 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:16:58.960945 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:58.960911 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:59.119146 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:59.119057 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:59.712678 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:59.712650 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:16:59.907920 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:16:59.907891 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:00.307448 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:00.307415 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:00.504342 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:00.504309 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:00.947981 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:00.947942 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:00.948158 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:00.947957 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:00.948158 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:00.948058 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:00.948158 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:00.948064 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:00.948158 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:00.948141 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:00.948333 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:00.948234 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:01.113023 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.112994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:01.468636 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.468596 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:01.660690 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.660661 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:01.755296 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.755226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:01.755296 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.755286 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:01.755461 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.755370 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:17:01.755461 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.755426 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs podName:e8a11aea-1720-432f-a0e4-63a9acbb1efa nodeName:}" failed. No retries permitted until 2026-11-01 14:17:33.75541218 +0000 UTC m=+66.282784907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs") pod "network-metrics-daemon-f72xc" (UID: "e8a11aea-1720-432f-a0e4-63a9acbb1efa") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 14:17:01.755461 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.755373 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 14:17:01.755552 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.755491 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret podName:99114e06-1d8b-4e06-aa86-4436de92e794 nodeName:}" failed. No retries permitted until 2026-11-01 14:17:33.755478798 +0000 UTC m=+66.282851536 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret") pod "global-pull-secret-syncer-ckfsz" (UID: "99114e06-1d8b-4e06-aa86-4436de92e794") : object "kube-system"/"original-pull-secret" not registered Nov 01 14:17:01.821539 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.821507 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:01.856491 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.856465 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:01.856614 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.856586 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 14:17:01.856614 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.856599 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 14:17:01.856614 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.856609 2583 projected.go:196] Error preparing data for projected volume kube-api-access-rczrt for pod openshift-network-diagnostics/network-check-target-t7sh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:17:01.856704 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:01.856649 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt podName:281050d7-54dd-4003-abe4-aa99c6c68a3a nodeName:}" failed. No retries permitted until 2026-11-01 14:17:33.856639246 +0000 UTC m=+66.384011970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczrt" (UniqueName: "kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt") pod "network-check-target-t7sh8" (UID: "281050d7-54dd-4003-abe4-aa99c6c68a3a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 14:17:01.974301 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:01.974273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:02.139542 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:02.139508 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:02.510171 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:02.510094 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:02.672777 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:02.672743 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:02.947648 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:02.947618 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:02.947808 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:02.947709 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:02.947808 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:02.947720 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:02.947808 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:02.947739 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:02.947905 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:02.947815 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:02.947905 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:02.947871 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:03.084513 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:03.084484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:03.308277 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:03.308250 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:03.712605 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:03.712530 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:03.909283 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:03.909254 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:04.115360 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:04.115331 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:04.276734 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:04.276703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:04.512233 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:04.512191 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:04.947785 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:04.947708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:04.948245 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:04.947789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:04.948245 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:04.947811 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:04.948245 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:04.947886 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:04.948245 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:04.948006 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:04.948245 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:04.948107 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:05.120473 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:05.120439 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:05.705073 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:05.705039 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:05.911283 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:05.911253 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:06.112603 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:06.112574 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:06.517281 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:06.517251 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:06.704913 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:06.704880 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:06.947571 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:06.947493 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:06.947571 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:06.947515 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:06.947744 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:06.947596 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:06.947744 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:06.947610 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:06.947744 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:06.947693 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:06.947744 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:06.947731 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:08.947570 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:08.947532 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:08.947959 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:08.947608 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:08.947959 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:08.947636 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:08.947959 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:08.947712 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:08.947959 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:08.947786 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:08.947959 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:08.947840 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:10.947349 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:10.947316 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:10.947714 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:10.947316 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:10.947714 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:10.947417 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:10.947714 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:10.947324 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:10.947714 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:10.947493 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:10.947714 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:10.947579 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:12.947391 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:12.947357 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:12.947858 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:12.947357 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:12.947858 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:12.947457 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:12.947858 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:12.947362 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:12.947858 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:12.947528 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:12.947858 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:12.947595 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:14.947155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:14.947119 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:14.947155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:14.947154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:14.947698 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:14.947160 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:14.947698 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:14.947296 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-t7sh8" podUID="281050d7-54dd-4003-abe4-aa99c6c68a3a" Nov 01 14:17:14.947698 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:14.947396 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckfsz" podUID="99114e06-1d8b-4e06-aa86-4436de92e794" Nov 01 14:17:14.947698 ip-10-0-131-109 kubenswrapper[2583]: E1101 14:17:14.947463 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f72xc" podUID="e8a11aea-1720-432f-a0e4-63a9acbb1efa" Nov 01 14:17:15.082907 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:15.082862 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:15.240134 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:15.240057 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:15.553085 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:15.553053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:15.707508 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:15.707451 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:16.307679 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.307656 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeReady" Nov 01 14:17:16.308077 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.307767 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 14:17:16.460439 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.460407 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fx4jc"] Nov 01 14:17:16.467923 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.467890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.469848 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.469823 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zx5dd"] Nov 01 14:17:16.472460 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.472441 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4279c"] Nov 01 14:17:16.472581 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.472562 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.475201 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.475184 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fx4jc"] Nov 01 14:17:16.475321 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.475235 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zx5dd"] Nov 01 14:17:16.475321 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.475300 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.479252 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.479231 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4279c"] Nov 01 14:17:16.520629 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.520606 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 14:17:16.520932 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.520917 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 14:17:16.521045 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.521031 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjzzz\"" Nov 01 14:17:16.524358 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.524339 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 14:17:16.524358 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.524353 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 14:17:16.524446 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.524345 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x2zjc\"" Nov 01 14:17:16.524446 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.524437 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 14:17:16.529740 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.529723 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8xv6q\"" Nov 01 14:17:16.529827 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.529815 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 14:17:16.529827 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.529821 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 14:17:16.529950 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.529938 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 14:17:16.530137 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.530120 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 14:17:16.564530 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564476 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-data-volume\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.564530 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvvx\" (UniqueName: \"kubernetes.io/projected/3880723f-671b-4e04-bfa6-132e59dd9145-kube-api-access-xtvvx\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.564530 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5df786-479c-485b-ba4e-255d1cc48cd5-cert\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.564686 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddc46\" (UniqueName: \"kubernetes.io/projected/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-api-access-ddc46\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.564686 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.564764 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564695 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3880723f-671b-4e04-bfa6-132e59dd9145-metrics-tls\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.564764 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-crio-socket\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.564764 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.564862 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564817 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3880723f-671b-4e04-bfa6-132e59dd9145-config-volume\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.564862 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564847 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880723f-671b-4e04-bfa6-132e59dd9145-tmp-dir\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.564930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.564863 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxnln\" (UniqueName: \"kubernetes.io/projected/9d5df786-479c-485b-ba4e-255d1cc48cd5-kube-api-access-xxnln\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.665308 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-data-volume\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.665308 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665311 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvvx\" (UniqueName: \"kubernetes.io/projected/3880723f-671b-4e04-bfa6-132e59dd9145-kube-api-access-xtvvx\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.665536 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665335 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5df786-479c-485b-ba4e-255d1cc48cd5-cert\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.665536 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665381 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddc46\" (UniqueName: \"kubernetes.io/projected/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-api-access-ddc46\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.665536 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665416 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.665687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3880723f-671b-4e04-bfa6-132e59dd9145-metrics-tls\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.665687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-crio-socket\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.665687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.665687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665659 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3880723f-671b-4e04-bfa6-132e59dd9145-config-volume\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.665687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665681 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-data-volume\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.665915 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880723f-671b-4e04-bfa6-132e59dd9145-tmp-dir\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.665915 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnln\" (UniqueName: \"kubernetes.io/projected/9d5df786-479c-485b-ba4e-255d1cc48cd5-kube-api-access-xxnln\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.665915 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665753 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-crio-socket\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.666055 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.665969 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880723f-671b-4e04-bfa6-132e59dd9145-tmp-dir\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.666131 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.666111 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.666324 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.666304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3880723f-671b-4e04-bfa6-132e59dd9145-config-volume\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.669250 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.669204 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.669363 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.669253 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3880723f-671b-4e04-bfa6-132e59dd9145-metrics-tls\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.669363 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.669312 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5df786-479c-485b-ba4e-255d1cc48cd5-cert\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.721923 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.721883 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddc46\" (UniqueName: \"kubernetes.io/projected/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-api-access-ddc46\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.722071 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.721998 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvvx\" (UniqueName: \"kubernetes.io/projected/3880723f-671b-4e04-bfa6-132e59dd9145-kube-api-access-xtvvx\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.722339 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.722323 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnln\" (UniqueName: \"kubernetes.io/projected/9d5df786-479c-485b-ba4e-255d1cc48cd5-kube-api-access-xxnln\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.784669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.784629 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:16.787707 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.787686 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:16.796406 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.796337 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:16.947296 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.947265 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:16.947296 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.947280 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:16.947296 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:16.947305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:17.000269 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000249 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 14:17:17.000379 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000326 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 14:17:17.000477 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000461 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 14:17:17.000543 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000521 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\"" Nov 01 14:17:17.000601 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000547 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 14:17:17.000601 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000593 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\"" Nov 01 14:17:17.000669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.000618 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\"" Nov 01 14:17:17.173728 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.173700 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4279c"] Nov 01 14:17:17.197457 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.197431 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4279c" event={"ID":"1ad903cc-65c9-4834-88c2-1ada8e4ae38c","Type":"ContainerStarted","Data":"31e7979ccd9c75d3682a055fe12f415bc37ef03585b302e3b73e261d36c97922"} Nov 01 14:17:17.214881 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:17.214849 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5df786_479c_485b_ba4e_255d1cc48cd5.slice/crio-b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2 WatchSource:0}: Error finding container b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2: Status 404 returned error can't find the container with id b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2 Nov 01 14:17:17.216533 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.216493 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zx5dd"] Nov 01 14:17:17.218727 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:17.218700 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880723f_671b_4e04_bfa6_132e59dd9145.slice/crio-5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011 WatchSource:0}: Error finding container 5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011: Status 404 returned error can't find the container with id 5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011 Nov 01 14:17:17.219521 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.219503 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fx4jc"] Nov 01 14:17:17.455281 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:17.455176 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:18.180897 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:18.180849 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:18.200731 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:18.200697 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4279c" event={"ID":"1ad903cc-65c9-4834-88c2-1ada8e4ae38c","Type":"ContainerStarted","Data":"d9dd3468ac79ea9b1d178327d57b708d52d84777f8e92b52372dfcafd978a041"} Nov 01 14:17:18.201820 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:18.201797 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zx5dd" event={"ID":"9d5df786-479c-485b-ba4e-255d1cc48cd5","Type":"ContainerStarted","Data":"b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2"} Nov 01 14:17:18.202787 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:18.202768 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fx4jc" event={"ID":"3880723f-671b-4e04-bfa6-132e59dd9145","Type":"ContainerStarted","Data":"5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011"} Nov 01 14:17:18.884669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:18.884556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:19.046067 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.046028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:19.203082 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.202971 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:19.207160 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.207126 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4279c" event={"ID":"1ad903cc-65c9-4834-88c2-1ada8e4ae38c","Type":"ContainerStarted","Data":"04045d899ffb2233baff868ac9551ac2878f787701f01d1b7cad20b97982a4b1"} Nov 01 14:17:19.236667 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.236634 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh"] Nov 01 14:17:19.240195 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.239686 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.287045 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.287011 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g56\" (UniqueName: \"kubernetes.io/projected/61db8c12-1910-483d-9ad4-8c491354c549-kube-api-access-r7g56\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.287200 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.287180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.287283 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.287225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.287373 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.287343 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61db8c12-1910-483d-9ad4-8c491354c549-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.292530 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.292505 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 14:17:19.292647 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.292577 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 14:17:19.295942 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.295920 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 14:17:19.295942 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.295926 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 14:17:19.296111 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.296043 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 14:17:19.296856 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.296837 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh"] Nov 01 14:17:19.357913 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.357884 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:19.387789 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.387757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.387960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.387808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61db8c12-1910-483d-9ad4-8c491354c549-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.387960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.387860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g56\" (UniqueName: \"kubernetes.io/projected/61db8c12-1910-483d-9ad4-8c491354c549-kube-api-access-r7g56\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.387960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.387947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.389463 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.389431 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61db8c12-1910-483d-9ad4-8c491354c549-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.391822 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.391792 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.391941 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.391828 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.393043 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.393014 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vshv8\"" Nov 01 14:17:19.446959 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.446927 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g56\" (UniqueName: \"kubernetes.io/projected/61db8c12-1910-483d-9ad4-8c491354c549-kube-api-access-r7g56\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.554869 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.554637 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:19.931435 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:19.931411 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh"] Nov 01 14:17:19.940307 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:19.940272 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61db8c12_1910_483d_9ad4_8c491354c549.slice/crio-f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae WatchSource:0}: Error finding container f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae: Status 404 returned error can't find the container with id f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae Nov 01 14:17:20.212301 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:20.212225 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fx4jc" event={"ID":"3880723f-671b-4e04-bfa6-132e59dd9145","Type":"ContainerStarted","Data":"55386b486d692f22a5be7e9249bd8782b4bfd705a8061c41807b4b3d0433444f"} Nov 01 14:17:20.212301 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:20.212289 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fx4jc" event={"ID":"3880723f-671b-4e04-bfa6-132e59dd9145","Type":"ContainerStarted","Data":"10e09068deb8d6efa908c6978c537aff2f112f624e97c0dab3fd64ce2c4e5513"} Nov 01 14:17:20.212504 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:20.212334 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:20.213579 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:20.213545 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" event={"ID":"61db8c12-1910-483d-9ad4-8c491354c549","Type":"ContainerStarted","Data":"f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae"} Nov 01 14:17:20.327880 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:20.327809 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-fx4jc" podStartSLOduration=1.5960432500000001 podStartE2EDuration="4.327790306s" totalImagesPullingTime="2.731747056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:20.326249716 +0000 UTC m=+52.853622463" watchObservedRunningTime="2026-11-01 14:17:20.327790306 +0000 UTC m=+52.855163057" Nov 01 14:17:21.219509 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:21.219460 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4279c" event={"ID":"1ad903cc-65c9-4834-88c2-1ada8e4ae38c","Type":"ContainerStarted","Data":"9320efc4758a6b8f80526825837515e664d3de1a5691aa494694f24570a0ada9"} Nov 01 14:17:21.221418 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:21.221385 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zx5dd" event={"ID":"9d5df786-479c-485b-ba4e-255d1cc48cd5","Type":"ContainerStarted","Data":"102119088cec439d81d0ac747592ee7bdd16f5cfa6ce3dd0caa797514933d04a"} Nov 01 14:17:21.333877 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:21.333819 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4279c" podStartSLOduration=1.986453678 podStartE2EDuration="5.333801963s" totalImagesPullingTime="3.347348285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 14:17:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:21.33351769 +0000 UTC m=+53.860890435" watchObservedRunningTime="2026-11-01 14:17:21.333801963 +0000 UTC m=+53.861174705" Nov 01 14:17:21.447261 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:21.447181 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zx5dd" podStartSLOduration=1.9425001370000001 podStartE2EDuration="5.447163762s" totalImagesPullingTime="3.504663625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:21.446258793 +0000 UTC m=+53.973631539" watchObservedRunningTime="2026-11-01 14:17:21.447163762 +0000 UTC m=+53.974536531" Nov 01 14:17:22.137239 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:22.137188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:22.226195 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:22.226158 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" event={"ID":"61db8c12-1910-483d-9ad4-8c491354c549","Type":"ContainerStarted","Data":"59a0ae9919b5352529df3f3d45897c5e9df2d2a3b909caf423e5f18a4be8ae20"} Nov 01 14:17:22.226734 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:22.226209 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" event={"ID":"61db8c12-1910-483d-9ad4-8c491354c549","Type":"ContainerStarted","Data":"af3edd5a33eeeedf936cf5ea38cf974e9e14210a25c3fe3ebbb991c71581f06e"} Nov 01 14:17:22.291617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:22.291590 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:22.340123 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:22.340067 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" podStartSLOduration=1.596151414 podStartE2EDuration="3.340053987s" totalImagesPullingTime="1.743902573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:22.339279638 +0000 UTC m=+54.866652383" watchObservedRunningTime="2026-11-01 14:17:22.340053987 +0000 UTC m=+54.867426732" Nov 01 14:17:24.244772 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:24.244735 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:24.979823 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:24.979787 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:25.187118 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.187083 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:25.348953 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.348921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:25.511190 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.511156 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:25.595564 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.595531 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pg8hh"] Nov 01 14:17:25.598825 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.598810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.637931 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.637903 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c54q\" (UniqueName: \"kubernetes.io/projected/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-kube-api-access-5c54q\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.637931 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.637932 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-sys\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638131 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.637951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638131 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.638032 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-root\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638131 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.638090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-textfile\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638131 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.638114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-wtmp\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638295 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.638133 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-tls\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638295 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.638150 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-metrics-client-ca\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.638295 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.638177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-accelerators-collector-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.674054 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.674029 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:25.738787 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738761 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-root\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.738911 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-textfile\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.738911 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738838 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-wtmp\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.738911 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-tls\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.738911 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738870 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-root\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.738911 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-metrics-client-ca\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-accelerators-collector-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c54q\" (UniqueName: \"kubernetes.io/projected/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-kube-api-access-5c54q\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.738999 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-sys\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.739005 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-wtmp\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.739023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739089 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.739051 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-sys\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739281 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.739160 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-textfile\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.739406 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.739390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-metrics-client-ca\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.763039 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.763014 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 14:17:25.763039 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.763014 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 14:17:25.763297 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.763071 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5wqpw\"" Nov 01 14:17:25.763297 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.763079 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 14:17:25.769551 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.769532 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-accelerators-collector-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.771266 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.771247 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.771426 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.771411 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-tls\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.794778 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.794755 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c54q\" (UniqueName: \"kubernetes.io/projected/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-kube-api-access-5c54q\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.913350 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:25.913287 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:25.923825 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:25.923776 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b6d85d_bd06_4e6c_9b9b_93fc59812d24.slice/crio-8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc WatchSource:0}: Error finding container 8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc: Status 404 returned error can't find the container with id 8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc Nov 01 14:17:26.032830 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:26.032803 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:26.136750 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:26.136718 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:26.239488 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:26.239406 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pg8hh" event={"ID":"03b6d85d-bd06-4e6c-9b9b-93fc59812d24","Type":"ContainerStarted","Data":"8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc"} Nov 01 14:17:26.297716 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:26.297688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:26.954907 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:26.954875 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:27.153142 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.153108 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:27.762995 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.762964 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:27.818888 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.818865 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:17:27.822152 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.822138 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.925370 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.925289 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 14:17:27.925370 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.925291 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 14:17:27.925370 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.925294 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 14:17:27.925607 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.925376 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 14:17:27.925607 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.925423 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 14:17:27.926079 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.926011 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 14:17:27.926207 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.926147 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 14:17:27.926376 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.926360 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vd72s\"" Nov 01 14:17:27.926615 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.926593 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 14:17:27.928266 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.928243 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:17:27.956543 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956512 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956543 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956579 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956600 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956729 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956761 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956899 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.956960 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956942 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.957234 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.956966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.957234 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.957039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.957234 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.957070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:27.976771 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:27.976745 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 14:17:28.057432 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057402 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057432 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057430 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057651 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057453 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057651 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057651 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057651 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057651 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057644 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057734 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057760 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.057883 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.057853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.059139 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.059111 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.059679 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.059655 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.060903 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.060571 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.060903 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.060571 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.060903 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.060639 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.060903 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.060705 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.060903 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.060738 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.061242 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.061177 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.061827 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.061808 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.062037 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.062016 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.062994 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.062974 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.063110 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.063090 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.117591 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.117558 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.145347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.145318 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:28.249202 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.249169 2583 generic.go:356] "Generic (PLEG): container finished" podID="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" containerID="9b87fb4a771e40294e209e145decebe9daeb366bbb2ad2e591cbb637477dc0bc" exitCode=0 Nov 01 14:17:28.249354 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.249233 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pg8hh" event={"ID":"03b6d85d-bd06-4e6c-9b9b-93fc59812d24","Type":"ContainerDied","Data":"9b87fb4a771e40294e209e145decebe9daeb366bbb2ad2e591cbb637477dc0bc"} Nov 01 14:17:28.608554 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:28.608312 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a72f134_3d8c_49b8_a3f8_b874e643ca0e.slice/crio-501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23 WatchSource:0}: Error finding container 501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23: Status 404 returned error can't find the container with id 501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23 Nov 01 14:17:28.609529 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.609500 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:17:28.645602 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.645580 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:28.805496 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.805467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:28.972417 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:28.972337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:29.254967 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.254877 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pg8hh" event={"ID":"03b6d85d-bd06-4e6c-9b9b-93fc59812d24","Type":"ContainerStarted","Data":"6726e113a5d41bda6a531d00b949bf811f06311fad6a4f02dc1ac83b90658c31"} Nov 01 14:17:29.254967 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.254935 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pg8hh" event={"ID":"03b6d85d-bd06-4e6c-9b9b-93fc59812d24","Type":"ContainerStarted","Data":"a13e937f0a2ca14830d6a401c19a6b35516d0fc74b1b4b36b45a0ba7ce2a2468"} Nov 01 14:17:29.256079 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.256054 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23"} Nov 01 14:17:29.370825 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.370765 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pg8hh" podStartSLOduration=2.64190477 podStartE2EDuration="4.370752685s" totalImagesPullingTime="1.663057637s" totalInitContainerRuntime="65.790278ms" isStatefulPod=true podCreationTimestamp="2026-11-01 14:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:29.369648287 +0000 UTC m=+61.897021032" watchObservedRunningTime="2026-11-01 14:17:29.370752685 +0000 UTC m=+61.898125451" Nov 01 14:17:29.582355 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.582326 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:29.743457 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.743423 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:29.900700 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:29.900622 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:30.223573 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.223499 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:30.346493 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.346467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:30.538510 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.538480 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-674fc8bc5b-hm9rs"] Nov 01 14:17:30.541540 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.541524 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.564060 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.564032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:30.595011 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.594990 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 14:17:30.595331 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.595298 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 14:17:30.595433 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.595355 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 14:17:30.595433 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.595367 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 14:17:30.595496 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.595471 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-uu6sl2qqfhq0\"" Nov 01 14:17:30.595544 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.595530 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-js46j\"" Nov 01 14:17:30.598499 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.598478 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-674fc8bc5b-hm9rs"] Nov 01 14:17:30.680900 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.680869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqg5\" (UniqueName: \"kubernetes.io/projected/b6a34a13-3327-4384-bc2d-09e30659af46-kube-api-access-4rqg5\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.681039 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.680910 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-client-certs\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.681039 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.680936 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a34a13-3327-4384-bc2d-09e30659af46-audit-log\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.681146 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.681046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-tls\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.681146 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.681087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-client-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.681146 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.681118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.681317 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.681210 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-metrics-server-audit-profiles\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781545 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781569 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-metrics-server-audit-profiles\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqg5\" (UniqueName: \"kubernetes.io/projected/b6a34a13-3327-4384-bc2d-09e30659af46-kube-api-access-4rqg5\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781622 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-client-certs\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781637 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a34a13-3327-4384-bc2d-09e30659af46-audit-log\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781687 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781667 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-tls\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.781930 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.781693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-client-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.782125 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.782098 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a34a13-3327-4384-bc2d-09e30659af46-audit-log\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.782336 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.782313 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.782618 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.782600 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-metrics-server-audit-profiles\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.784226 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.784195 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-client-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.784609 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.784590 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-tls\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.784760 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.784740 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-client-certs\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.838302 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.838246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqg5\" (UniqueName: \"kubernetes.io/projected/b6a34a13-3327-4384-bc2d-09e30659af46-kube-api-access-4rqg5\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:30.852191 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:30.852175 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:31.256678 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:31.256640 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a34a13_3327_4384_bc2d_09e30659af46.slice/crio-a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a WatchSource:0}: Error finding container a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a: Status 404 returned error can't find the container with id a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a Nov 01 14:17:31.258320 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:31.258300 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-674fc8bc5b-hm9rs"] Nov 01 14:17:31.263895 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:31.263870 2583 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" exitCode=0 Nov 01 14:17:31.263971 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:31.263946 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} Nov 01 14:17:31.265347 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:31.265322 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" event={"ID":"b6a34a13-3327-4384-bc2d-09e30659af46","Type":"ContainerStarted","Data":"a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a"} Nov 01 14:17:32.598380 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:32.598341 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:32.758453 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:32.758421 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:33.297482 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.297410 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:33.462848 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.462822 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:33.665486 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.665400 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:33.805745 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.805670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:33.805745 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.805725 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:33.884085 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.884054 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 14:17:33.884273 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.884176 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 14:17:33.888416 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.888387 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:33.888520 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.888396 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:33.894440 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.894416 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:33.906750 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.906721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:33.958065 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:33.958046 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 14:17:34.018874 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.018850 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 14:17:34.030210 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.030190 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:34.054327 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.054306 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:34.110573 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.110508 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\"" Nov 01 14:17:34.117634 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.117614 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\"" Nov 01 14:17:34.119719 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.119703 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:34.120133 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.120118 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\"" Nov 01 14:17:34.127531 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.127511 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:34.129365 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.129349 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:34.215099 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.214941 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:34.284286 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.284254 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} Nov 01 14:17:34.284425 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.284318 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} Nov 01 14:17:34.284425 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.284355 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} Nov 01 14:17:34.284425 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.284383 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} Nov 01 14:17:34.284425 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.284405 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} Nov 01 14:17:34.285796 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.285772 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" event={"ID":"b6a34a13-3327-4384-bc2d-09e30659af46","Type":"ContainerStarted","Data":"8e3f11dcad3171f5ab30e2d830392399bfbded62acd79a8578e247d99d497a1e"} Nov 01 14:17:34.403396 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.403299 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" podStartSLOduration=2.123142636 podStartE2EDuration="4.403286476s" totalImagesPullingTime="2.28014384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:34.401848139 +0000 UTC m=+66.929220884" watchObservedRunningTime="2026-11-01 14:17:34.403286476 +0000 UTC m=+66.930659225" Nov 01 14:17:34.437317 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.437285 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:17:34.443096 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.443071 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.450410 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.450386 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:34.496917 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.496882 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 14:17:34.497062 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.496999 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 14:17:34.497062 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.497047 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 14:17:34.497202 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.497047 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-29vjcn8bfe0n4\"" Nov 01 14:17:34.497202 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.497087 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 14:17:34.497307 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.497201 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 14:17:34.497307 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.497123 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 14:17:34.499634 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.499612 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 14:17:34.504825 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.504802 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ckfsz"] Nov 01 14:17:34.505944 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:34.505915 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99114e06_1d8b_4e06_aa86_4436de92e794.slice/crio-44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d WatchSource:0}: Error finding container 44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d: Status 404 returned error can't find the container with id 44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.513988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514054 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514077 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514155 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514197 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514230 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514359 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514420 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514619 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514491 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.514822 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.514624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.519707 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.519688 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 14:17:34.519882 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.519868 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 14:17:34.552456 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:34.552420 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281050d7_54dd_4003_abe4_aa99c6c68a3a.slice/crio-d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55 WatchSource:0}: Error finding container d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55: Status 404 returned error can't find the container with id d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55 Nov 01 14:17:34.553504 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:34.553451 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a11aea_1720_432f_a0e4_63a9acbb1efa.slice/crio-242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299 WatchSource:0}: Error finding container 242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299: Status 404 returned error can't find the container with id 242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299 Nov 01 14:17:34.597927 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.597886 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 14:17:34.598075 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.598015 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 14:17:34.598329 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.598305 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 14:17:34.598489 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.598471 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 14:17:34.598571 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.598490 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 14:17:34.598571 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.598554 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvptn\"" Nov 01 14:17:34.600967 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.600943 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-t7sh8"] Nov 01 14:17:34.601056 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.601006 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f72xc"] Nov 01 14:17:34.615949 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.615924 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.615955 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.615977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616004 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616046 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616046 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616144 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616267 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616258 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616313 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616358 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616979 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616979 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616864 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.616979 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616904 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.617129 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.616983 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.617749 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.617721 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.617840 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.617777 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.619113 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.619084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.619894 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.619477 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.619894 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.619544 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.620088 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.620063 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.620655 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.620344 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.620655 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.620412 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.620655 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.620609 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.620655 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.620638 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.621033 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.621013 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.621532 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.621516 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.621617 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.621519 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.622011 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.621989 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.622340 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.622321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.622467 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.622448 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.662466 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.662395 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:17:34.675882 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.675861 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:34.794827 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:34.794788 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:35.210242 ip-10-0-131-109 kubenswrapper[2583]: W1101 14:17:35.210198 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8121105_7c56_42bf_89a2_858453626862.slice/crio-9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f WatchSource:0}: Error finding container 9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f: Status 404 returned error can't find the container with id 9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f Nov 01 14:17:35.211871 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.211850 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:17:35.290790 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.290757 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerStarted","Data":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} Nov 01 14:17:35.290918 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.290839 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerStarted","Data":"9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f"} Nov 01 14:17:35.294400 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.294373 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerStarted","Data":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} Nov 01 14:17:35.297669 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.297603 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ckfsz" event={"ID":"99114e06-1d8b-4e06-aa86-4436de92e794","Type":"ContainerStarted","Data":"44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d"} Nov 01 14:17:35.299318 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.299295 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f72xc" event={"ID":"e8a11aea-1720-432f-a0e4-63a9acbb1efa","Type":"ContainerStarted","Data":"242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299"} Nov 01 14:17:35.300367 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.300342 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7sh8" event={"ID":"281050d7-54dd-4003-abe4-aa99c6c68a3a","Type":"ContainerStarted","Data":"d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55"} Nov 01 14:17:35.542203 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:35.542134 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.310297097 podStartE2EDuration="8.542114245s" totalImagesPullingTime="5.220044564s" totalInitContainerRuntime="11.772584ms" isStatefulPod=false podCreationTimestamp="2026-11-01 14:17:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:35.540210565 +0000 UTC m=+68.067583310" watchObservedRunningTime="2026-11-01 14:17:35.542114245 +0000 UTC m=+68.069486990" Nov 01 14:17:36.306686 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:36.306656 2583 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" exitCode=0 Nov 01 14:17:36.308189 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:36.308161 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} Nov 01 14:17:36.320763 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:36.320740 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:36.482037 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:36.481994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:36.651726 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:36.651651 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:37.132572 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:37.132539 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:37.312029 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:37.311991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:37.314529 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:37.314493 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f72xc" event={"ID":"e8a11aea-1720-432f-a0e4-63a9acbb1efa","Type":"ContainerStarted","Data":"3a242756b38c97e75193676e7e45aac65d70b7473ec4b32f95fe26489e05fd11"} Nov 01 14:17:37.314678 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:37.314623 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f72xc" event={"ID":"e8a11aea-1720-432f-a0e4-63a9acbb1efa","Type":"ContainerStarted","Data":"af7e2eda85784c6d1476a53d17614fb5c9847029d81cc9b1d2f5eb6a79989363"} Nov 01 14:17:37.429763 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:37.429660 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f72xc" podStartSLOduration=67.68602608 podStartE2EDuration="1m9.429640363s" totalImagesPullingTime="1.743614283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:37.428168124 +0000 UTC m=+69.955540897" watchObservedRunningTime="2026-11-01 14:17:37.429640363 +0000 UTC m=+69.957013110" Nov 01 14:17:37.486944 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:37.486909 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:38.044650 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:38.044616 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:38.222237 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:38.222181 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:39.327005 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:39.326967 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-t7sh8" event={"ID":"281050d7-54dd-4003-abe4-aa99c6c68a3a","Type":"ContainerStarted","Data":"015bedba029f8a406e5baf4fd77e21a4fd260591a40c7874580c12e25d54d4ad"} Nov 01 14:17:39.327877 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:39.327854 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:39.443059 ip-10-0-131-109 kubenswrapper[2583]: I1101 14:17:39.442957 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-t7sh8" podStartSLOduration=66.904629467 podStartE2EDuration="1m11.44293919s" totalImagesPullingTime="4.538309723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:17:39.441988089 +0000 UTC m=+71.969360829" watchObservedRunningTime="2026-11-01 14:17:39.44293919 +0000 UTC m=+71.970312038" Nov 01 14:17:40.849962 ip-10-0-131-109 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 14:17:40.869399 ip-10-0-131-109 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 14:17:40.869632 ip-10-0-131-109 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 14:17:40.869696 ip-10-0-131-109 systemd[1]: kubelet.service: Consumed 4.108s CPU time. Nov 01 14:17:40.889735 ip-10-0-131-109 systemd[1]: Starting Kubernetes Kubelet... Nov 01 14:17:40.980714 ip-10-0-131-109 kubenswrapper[5473]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 14:17:40.980714 ip-10-0-131-109 kubenswrapper[5473]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 14:17:40.980714 ip-10-0-131-109 kubenswrapper[5473]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 14:17:40.980714 ip-10-0-131-109 kubenswrapper[5473]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 14:17:40.982888 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982800 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:40.982888 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982889 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982893 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982896 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982899 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982903 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982905 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982908 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982910 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982913 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982916 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982919 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982921 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982924 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982927 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982929 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982932 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982935 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982937 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982940 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982942 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:40.982966 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982945 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982948 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982950 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982953 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982956 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982959 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982961 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982964 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982971 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982974 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982977 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982980 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982983 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982986 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982988 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982991 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982994 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982996 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.982999 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983002 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:40.983479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983004 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983007 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983010 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983013 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983015 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983018 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983021 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983024 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983026 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983029 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983032 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983034 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983037 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983042 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983046 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983050 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983054 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983057 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983060 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:40.984055 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983063 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983066 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983069 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983072 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983075 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983078 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983081 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983083 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983086 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983089 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983092 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983094 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983097 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983100 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983102 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983105 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983107 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983110 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983113 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:40.984631 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983116 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983119 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983121 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983125 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983129 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983132 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983136 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983138 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983141 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983144 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983147 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983149 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983152 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983155 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983158 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983161 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983164 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983166 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:40.985139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983169 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983172 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983175 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983178 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983181 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983184 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983187 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983189 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.983192 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984170 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984179 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984182 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984185 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984189 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984193 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984197 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984199 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984202 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984205 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984207 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:40.985801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984210 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984229 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984235 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984239 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984243 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984245 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984248 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984251 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984254 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984257 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984259 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984262 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984265 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984268 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984270 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984273 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984277 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984280 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984283 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:40.986398 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984285 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984288 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984290 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984294 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984298 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984301 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984304 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984307 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984311 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984314 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984316 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984319 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984321 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984324 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984327 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984330 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984332 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984335 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984337 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:40.986932 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984340 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984343 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984345 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984348 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984350 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984353 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984356 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984358 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984361 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984363 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984367 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984370 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984373 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984375 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984378 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984380 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984383 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984386 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984388 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984391 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:40.987498 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984395 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984399 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984405 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984408 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984410 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984413 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984418 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984421 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984424 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984426 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984429 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984432 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984435 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984438 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984440 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984443 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984445 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984448 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984451 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984454 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:40.988042 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984456 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984459 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984462 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984465 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984468 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984470 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984473 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984476 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984479 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984482 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984484 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984487 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984489 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984492 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984494 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984498 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.984500 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985109 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985119 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985122 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:40.988574 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985125 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985128 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985131 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985133 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985136 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985139 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985142 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985145 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985148 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985151 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985154 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985159 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985163 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985167 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985172 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985176 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985180 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985183 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985185 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:40.989053 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985188 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985191 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985194 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985196 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985199 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985202 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985205 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985208 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985226 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985231 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985235 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985242 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985246 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985250 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985255 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985259 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985263 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985266 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985269 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:40.989577 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985272 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985275 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985278 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985280 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985283 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985285 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985288 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985291 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985293 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985296 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985298 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985301 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985304 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985307 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985310 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985312 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985315 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985318 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985323 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:40.990049 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985327 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985330 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985333 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985337 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985340 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985344 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985348 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985352 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985357 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985361 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985365 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985369 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985373 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985375 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985378 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985381 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985383 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985386 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985388 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:40.990570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985391 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985394 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985396 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985400 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985406 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985410 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985413 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985416 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985418 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985421 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985424 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985427 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985429 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985432 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985436 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985440 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985444 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985448 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985451 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985454 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:40.991057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985456 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985459 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985462 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985465 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985468 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985472 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.985476 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985605 5473 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985619 5473 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985628 5473 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985634 5473 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985641 5473 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985647 5473 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985654 5473 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985660 5473 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985664 5473 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985670 5473 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985674 5473 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985678 5473 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985681 5473 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985685 5473 flags.go:64] FLAG: --cgroup-root="" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985688 5473 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 14:17:40.991568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985691 5473 flags.go:64] FLAG: --client-ca-file="" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985694 5473 flags.go:64] FLAG: --cloud-provider="external" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985698 5473 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985705 5473 flags.go:64] FLAG: --cluster-domain="" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985708 5473 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985711 5473 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985715 5473 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985718 5473 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985723 5473 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985726 5473 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985729 5473 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985733 5473 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985736 5473 flags.go:64] FLAG: --contention-profiling="false" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985739 5473 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985743 5473 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985746 5473 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985749 5473 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985753 5473 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985757 5473 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985760 5473 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985763 5473 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985766 5473 flags.go:64] FLAG: --enable-server="true" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985769 5473 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985774 5473 flags.go:64] FLAG: --event-burst="100" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985777 5473 flags.go:64] FLAG: --event-qps="50" Nov 01 14:17:40.992181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985780 5473 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985785 5473 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985788 5473 flags.go:64] FLAG: --eviction-hard="" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985793 5473 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985796 5473 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985799 5473 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985803 5473 flags.go:64] FLAG: --eviction-soft="" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985806 5473 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985810 5473 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985814 5473 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985818 5473 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985821 5473 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985824 5473 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985827 5473 flags.go:64] FLAG: --feature-gates="" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985831 5473 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985835 5473 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985839 5473 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985842 5473 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985846 5473 flags.go:64] FLAG: --healthz-port="10248" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985849 5473 flags.go:64] FLAG: --help="false" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985852 5473 flags.go:64] FLAG: --hostname-override="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985856 5473 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985859 5473 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 14:17:40.992922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985862 5473 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985865 5473 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985869 5473 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985872 5473 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985875 5473 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985878 5473 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985881 5473 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985884 5473 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985887 5473 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985890 5473 flags.go:64] FLAG: --kube-reserved="" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985894 5473 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985897 5473 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985900 5473 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985904 5473 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985907 5473 flags.go:64] FLAG: --lock-file="" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985910 5473 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985913 5473 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985916 5473 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985923 5473 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985926 5473 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985929 5473 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985932 5473 flags.go:64] FLAG: --logging-format="text" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985936 5473 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985939 5473 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 14:17:40.993504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985942 5473 flags.go:64] FLAG: --manifest-url="" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985946 5473 flags.go:64] FLAG: --manifest-url-header="" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985951 5473 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985954 5473 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985959 5473 flags.go:64] FLAG: --max-pods="110" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985962 5473 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985966 5473 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985969 5473 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985972 5473 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985975 5473 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985978 5473 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985981 5473 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985989 5473 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985992 5473 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985995 5473 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.985998 5473 flags.go:64] FLAG: --pod-cidr="" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986001 5473 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986004 5473 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986009 5473 flags.go:64] FLAG: --pods-per-core="0" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986012 5473 flags.go:64] FLAG: --port="10250" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986015 5473 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986018 5473 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0396536c18092b566" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986022 5473 flags.go:64] FLAG: --qos-reserved="" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986025 5473 flags.go:64] FLAG: --read-only-port="10255" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986028 5473 flags.go:64] FLAG: --register-node="true" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986031 5473 flags.go:64] FLAG: --register-with-taints="" Nov 01 14:17:40.994120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986037 5473 flags.go:64] FLAG: --registry-burst="10" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986040 5473 flags.go:64] FLAG: --registry-qps="5" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986043 5473 flags.go:64] FLAG: --reserved-cpus="" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986046 5473 flags.go:64] FLAG: --reserved-memory="" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986050 5473 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986053 5473 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986057 5473 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986060 5473 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986063 5473 flags.go:64] FLAG: --runonce="false" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986067 5473 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986070 5473 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986073 5473 flags.go:64] FLAG: --seccomp-default="false" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986076 5473 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986079 5473 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986082 5473 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986086 5473 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986089 5473 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986092 5473 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986096 5473 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986099 5473 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986101 5473 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986105 5473 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986108 5473 flags.go:64] FLAG: --system-cgroups="" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986111 5473 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986122 5473 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 14:17:40.994789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986125 5473 flags.go:64] FLAG: --tls-cert-file="" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986128 5473 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986133 5473 flags.go:64] FLAG: --tls-min-version="" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986136 5473 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986139 5473 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986142 5473 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986146 5473 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986150 5473 flags.go:64] FLAG: --v="2" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986157 5473 flags.go:64] FLAG: --version="false" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986163 5473 flags.go:64] FLAG: --vmodule="" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986169 5473 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986174 5473 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986410 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986417 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986421 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986424 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986427 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986430 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986433 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986436 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986439 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986441 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986444 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:40.995451 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986447 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986450 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986454 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986458 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986462 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986466 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986469 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986472 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986477 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986480 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986483 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986486 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986488 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986491 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986494 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986497 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986502 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986505 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986507 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:40.996586 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986510 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986513 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986515 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986518 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986521 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986524 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986527 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986529 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986532 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986535 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986538 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986541 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986544 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986546 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986550 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986552 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986555 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986557 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986560 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986563 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:40.997622 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986566 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986570 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986572 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986575 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986578 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986581 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986583 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986586 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986588 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986592 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986595 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986598 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986600 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986603 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986606 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986608 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986611 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986614 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986616 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:40.998192 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986619 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986622 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986625 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986628 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986631 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986633 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986636 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986639 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986641 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986644 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986646 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986649 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986652 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986654 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986690 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986694 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986696 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986699 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986702 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986722 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:40.998756 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986725 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986728 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986733 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986735 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986738 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986741 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986744 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986748 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986752 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986755 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986758 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986761 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986763 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986766 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986769 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986773 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:40.999706 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986776 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.986806 5473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986849 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986853 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986856 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986859 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986862 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986865 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986868 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986871 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986875 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986877 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986880 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986883 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986886 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986888 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:41.000453 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986891 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986894 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986896 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986900 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986903 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986906 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986908 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986917 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986920 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986922 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986925 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986928 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986930 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986933 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986936 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986939 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986942 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986945 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986948 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:41.001100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986951 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986954 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986956 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986959 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986962 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986964 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986967 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986971 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986974 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986977 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986979 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986982 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986985 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986987 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986990 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986993 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986996 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.986999 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987002 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:41.001808 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987004 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987007 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987010 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987012 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987015 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987017 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987020 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987022 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987025 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987028 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987031 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987036 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987040 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987045 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987049 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987055 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987061 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987066 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987070 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987072 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:41.002495 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987077 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987080 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987082 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987085 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987088 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987090 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987093 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987096 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987098 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987102 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987105 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987108 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987110 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987113 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987116 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987118 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987121 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987123 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987126 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:41.003004 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987130 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987134 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987137 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987141 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987144 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987147 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987150 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987153 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987156 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987158 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987161 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987164 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987168 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987172 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.987174 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.991082 5473 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.991099 5473 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.991114 5473 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.991119 5473 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 14:17:41.003500 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991175 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991181 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991185 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991189 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991192 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991195 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991197 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991200 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991203 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991206 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991209 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991228 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991232 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991236 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991239 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991241 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991244 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991247 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991250 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991253 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:41.003953 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991256 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991259 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991262 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991264 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991267 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991270 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991273 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991275 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991278 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991283 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991286 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991289 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991292 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991295 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991297 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991300 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991303 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991306 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991309 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991312 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:41.004520 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991315 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991317 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991320 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991323 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991325 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991330 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991334 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991338 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991341 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991344 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991346 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991349 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991352 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991355 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991358 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991361 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991363 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991366 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991369 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:41.005103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991372 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991375 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991379 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991382 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991385 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991388 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991390 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991393 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991396 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991398 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991401 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991404 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991406 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991409 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991412 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991414 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991417 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991420 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991423 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991425 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:41.005740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991428 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991432 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991434 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991437 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991440 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991442 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991445 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991448 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991450 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991453 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991455 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991458 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991461 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991464 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991467 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991469 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991473 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991475 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991478 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991480 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:41.006521 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991483 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991485 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991488 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991491 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991493 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991496 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991499 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.991528 5473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991670 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991675 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991678 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991681 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991685 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991688 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991691 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:41.007184 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991694 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991697 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991700 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991702 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991705 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991708 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991710 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991713 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991715 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991718 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991721 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991724 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991726 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991729 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991732 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991735 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991738 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991740 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991743 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:41.007603 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991745 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991748 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991750 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991754 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991759 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991762 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991765 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991768 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991771 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991774 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991778 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991780 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991783 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991786 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991788 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991791 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991794 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991796 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991799 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991801 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:41.008177 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991804 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991807 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991809 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991812 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991815 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991817 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991819 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991822 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991825 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991828 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991830 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991833 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991835 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991838 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991841 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991843 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991846 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991849 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991851 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:41.008874 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991854 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991857 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991859 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991862 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991865 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991867 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991870 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991873 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991875 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991878 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991881 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991883 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991886 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991888 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991891 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991893 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991896 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991898 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991901 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991904 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:41.009493 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991907 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991909 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991912 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991915 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991919 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991922 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991925 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991928 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991930 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991933 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991936 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991939 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991942 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991944 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991947 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991950 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991952 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991955 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991958 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991960 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:41.010259 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.991963 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.991985 5473 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992031 5473 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992035 5473 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992038 5473 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992041 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992044 5473 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992046 5473 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992049 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992052 5473 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992055 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992057 5473 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992060 5473 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992063 5473 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992065 5473 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992069 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 14:17:41.010918 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992071 5473 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992074 5473 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992077 5473 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992082 5473 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992086 5473 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992090 5473 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992094 5473 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992098 5473 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992102 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992107 5473 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992112 5473 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992119 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992124 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992129 5473 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992133 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992137 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992141 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992145 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992149 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 14:17:41.011411 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992154 5473 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992159 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992163 5473 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992167 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992171 5473 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992176 5473 feature_gate.go:386] unrecognized feature gate: Example Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992182 5473 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992188 5473 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992193 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992198 5473 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992203 5473 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992207 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992227 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992234 5473 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992238 5473 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992242 5473 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992245 5473 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992249 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992254 5473 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992258 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 14:17:41.011922 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992262 5473 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992266 5473 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992270 5473 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992274 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992280 5473 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992284 5473 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992288 5473 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992292 5473 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992296 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992301 5473 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992305 5473 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992309 5473 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992314 5473 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992318 5473 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992322 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992327 5473 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992331 5473 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992336 5473 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992341 5473 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 14:17:41.012541 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992345 5473 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992349 5473 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992353 5473 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992357 5473 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992361 5473 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992365 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992370 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992374 5473 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992378 5473 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992384 5473 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992388 5473 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992392 5473 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992397 5473 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992401 5473 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992405 5473 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992409 5473 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992413 5473 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992420 5473 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992423 5473 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992426 5473 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 14:17:41.013057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992428 5473 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992431 5473 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992434 5473 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992436 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992439 5473 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992442 5473 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992445 5473 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992449 5473 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992454 5473 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992457 5473 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992460 5473 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992463 5473 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992466 5473 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:17:40.992470 5473 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.992566 5473 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.993583 5473 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.993635 5473 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.993818 5473 server.go:1014] "Starting client certificate rotation" Nov 01 14:17:41.013974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.993919 5473 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.993974 5473 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 14:11:28 +0000 UTC" deadline="2028-05-13 06:17:32.718166269 +0000 UTC" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.994015 5473 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13407h59m51.724153665s" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.994333 5473 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.995450 5473 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:40.998487 5473 log.go:25] "Validated CRI v1 runtime API" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.001087 5473 log.go:25] "Validated CRI v1 image API" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.001940 5473 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 14:17:41.014813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.004183 5473 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99da58a5-bd91-45f0-ada5-0ca15684927b:/dev/nvme0n1p4 dd963982-9278-49f7-8cc4-4937440009e8:/dev/nvme0n1p3] Nov 01 14:17:41.015391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.004207 5473 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31e7979ccd9c75d3682a055fe12f415bc37ef03585b302e3b73e261d36c97922/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31e7979ccd9c75d3682a055fe12f415bc37ef03585b302e3b73e261d36c97922/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc/userdata/shm major:0 minor:271 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55/userdata/shm major:0 minor:345 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~projected/kube-api-access-5c54q:{mountpoint:/var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~projected/kube-api-access-5c54q major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volumes/kubernetes.io~projected/kube-api-access-xbqx8:{mountpoint:/var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volumes/kubernetes.io~projected/kube-api-access-xbqx8 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ad903cc-65c9-4834-88c2-1ada8e4ae38c/volumes/kubernetes.io~projected/kube-api-access-ddc46:{mountpoint:/var/lib/kubelet/pods/1ad903cc-65c9-4834-88c2-1ada8e4ae38c/volumes/kubernetes.io~projected/kube-api-access-ddc46 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ad903cc-65c9-4834-88c2-1ada8e4ae38c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/1ad903cc-65c9-4834-88c2-1ada8e4ae38c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/281050d7-54dd-4003-abe4-aa99c6c68a3a/volumes/kubernetes.io~projected/kube-api-access-rczrt:{mountpoint:/var/lib/kubelet/pods/281050d7-54dd-4003-abe4-aa99c6c68a3a/volumes/kubernetes.io~projected/kube-api-access-rczrt major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35cb9899-1884-4d74-8951-1a1c4a56b9e9/volumes/kubernetes.io~projected/kube-api-access-mx4cj:{mountpoint:/var/lib/kubelet/pods/35cb9899-1884-4d74-8951-1a1c4a56b9e9/volumes/kubernetes.io~projected/kube-api-access-mx4cj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3880723f-671b-4e04-bfa6-132e59dd9145/volumes/kubernetes.io~projected/kube-api-access-xtvvx:{mountpoint:/var/lib/kubelet/pods/3880723f-671b-4e04-bfa6-132e59dd9145/volumes/kubernetes.io~projected/kube-api-access-xtvvx major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3880723f-671b-4e04-bfa6-132e59dd9145/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3880723f-671b-4e04-bfa6-132e59dd9145/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~projected/kube-api-access-r7g56:{mountpoint:/var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~projected/kube-api-access-r7g56 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~projected/kube-api-access-pqzgz:{mountpoint:/var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~projected/kube-api-access-pqzgz major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/717175a0-02eb-40d7-b696-828a3cb7fcaa/volumes/kubernetes.io~projected/kube-api-access-clvll:{mountpoint:/var/lib/kubelet/pods/717175a0-02eb-40d7-b696-828a3cb7fcaa/volumes/kubernetes.io~projected/kube-api-access-clvll major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80966b44-0240-48ab-8528-3ebcfa9c2c12/volumes/kubernetes.io~projected/kube-api-access-xnj8k:{mountpoint:/var/lib/kubelet/pods/80966b44-0240-48ab-8528-3ebcfa9c2c12/volumes/kubernetes.io~projected/kube-api-access-xnj8k major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/alertmanager/9 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/config-reloader/6 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~projected/kube-api-access-pd2z4:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~projected/kube-api-access-pd2z4 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~projected/tls-assets major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/config-volume major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/web-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99114e06-1d8b-4e06-aa86-4436de92e794/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/99114e06-1d8b-4e06-aa86-4436de92e794/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d5df786-479c-485b-ba4e-255d1cc48cd5/volumes/kubernetes.io~projected/kube-api-access-xxnln:{mountpoint:/var/lib/kubelet/pods/9d5df786-479c-485b-ba4e-255d1cc48cd5/volumes/kubernetes.io~projected/kube-api-access-xxnln major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d5df786-479c-485b-ba4e-255d1cc48cd5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9d5df786-479c-485b-ba4e-255d1cc48cd5/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b241b884-12de-43ee-af05-7a92bff07b42/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b241b884-12de-43ee-af05-7a92bff07b42/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d/volumes/kubernetes.io~projected/kube-api-access-wjp2v:{mountpoint:/var/lib/kubelet/pods/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d/volumes/kubernetes.io~projected/kube-api-access-wjp2v major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~projected/kube-api-access-4rqg5:{mountpoint:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~projected/kube-api-access-4rqg5 major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/config-reloader/2 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/prometheus/15 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~empty-dir/config-out major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~projected/kube-api-access-p9pdt:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~projected/kube-api-access-p9pdt major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~projected/tls-assets major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/config major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/web-config major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dabe07b7-a91c-42ef-aa55-2d2783da8093/volumes/kubernetes.io~projected/kube-api-access-gsqpb:{mountpoint:/var/lib/kubelet/pods/dabe07b7-a91c-42ef-aa55-2d2783da8093/volumes/kubernetes.io~projected/kube-api-access-gsqpb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8a11aea-1720-432f-a0e4-63a9acbb1efa/volumes/kubernetes.io~projected/kube-api-access-fj9vt:{mountpoint:/var/lib/kubelet/pods/e8a11aea-1720-432f-a0e4-63a9acbb1efa/volumes/kubernetes.io~projected/kube-api-access-fj9vt major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8a11aea-1720-432f-a0e4-63a9acbb1efa/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e8a11aea-1720-432f-a0e4-63a9acbb1efa/volumes/kubernetes.io~secret/metrics-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5373225-4d3f-451e-ab69-45fee9340973/volumes/kubernetes.io~projected/kube-api-access-54vgg:{mountpoint:/var/lib/kubelet/pods/f5373225-4d3f-451e-ab69-45fee9340973/volumes/kubernetes.io~projected/kube-api-access-54vgg major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d7d06a649e7a4d082c3cb41a45ef80ed6b50a0ef4e19c5ff12b3ab56f840f741/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/43bd70b20bf227c4b39917d3a0a4e41f936bee64489a7778d0f6383a67583b31/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/774ec6dbb35f8fbcded3c80ab2bd30d2f253bbd267431fa6c99cb08a8c845067/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/76edd1e7aed769933e05046bd8c048250343acd8ffa793db84f54a41ba87e1b0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/40627a433bb5ac01fa9645333cf5e8fc8eada211aba1260a86ea93fe02310c9a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/89c3eed43a160571a0f440774b5997d0e2c6dd9ec94db44a25e6fa86138d1146/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/67e67f82470d8a095850237748843a4800b56159416bef7435d873ae804e5e3d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c31b64b102113d06e6ee3a1755e80838076ec3fc535f818bdc23890952498e18/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/587f67cd374c05873f0e1d3ac4254fef53c18ee72f08d6eda0338a042052cc54/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ed52afe1c3317ccfe60e787e89c80051bdf6d7a41a778753106487316e009748/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/a667e58f153dbbc8aabad636fb2e598a1f8578b721e04c5486b4069c108d026a/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/ea23bcb34389b5f54f0b42e4ecf9af9b1de04f52e293609c65f3d2290c0f61c4/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/677037b8c047bf1bc94cf881db427e73fb02804bc145d98c2422ef458d3919c3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ce4a4c3bd915dd386c7dce6dcd3fe251eccb7f929e3fc421f7e25168fbd6dd9b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d79e026f024fbe165af2bb876685dfb8b9558a2ff553d6517b7d49965862ff20/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0a06007bd86a269496ad60c1a7c1417f1515beb6978dfcb5c944ca71d0131172/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/0c43993bdc20d47b7cc63fb9c70b06a49ef1c97af5b2d9c3cd2aec2f59eafab4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d22b83bcc0d0bf209ee48de1634321cb9abdc7201b659b79b06967a123ce78e9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d6a534cd6f2fbb965b557199c32a8998e21a93899f70edae0ef2b0fc682d224b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3874bc7d417aa2d020de67a8ea2a42fe4e8c54428ca9d9bed6cb80b8c5274fe1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/039c5b03b3b082da1c38f008076b6d2c99a6d02fd8b178e0a1a5243877fdfd0e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/376ea62f78d782335aa727d59898f293d303190434870c2477fbdc5376895785/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b80e0e25439a4e19ab7ae9c01f9d2e1cb8c501de6297b65a9b55ee5a046ff68e/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/04c5d29d435b5d142f35c3d50c0c259e80c8168977490a280e4c82ec4b29a620/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/10471e4a7e2420251a6f2f138200d123ae57a5d0c9b8a1d4336e4a4bbbe940fd/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/6c32f0bab8534fa4be7e14588b68f5b7964d5f38a75d37d7c78af51e64141207/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/3ea27a03143f659a0f3f84faf874b2f5c07ee4014740291280f6c8674e4f2012/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/84de54ee166e8f45efa25dfd64db420d96d8f77e0e0b119794222c673457307c/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/960717f3959a29ec0295a5d0166573c937341cf4fff576d450d82efc05b0066e/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/0fa37bcb0e6e11e035e5dcd50c6113fe2139503b18c87aa3e8a804e3dacff98d/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0bc6224421bf2693fd4c9d5a785988984ed2e826da4254ef9deda4d93415cc9a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/29a8b89bc2c0a12f23c06c500770593395733a13ecf835e2ee5322e0eff42155/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/1aa36ee9c6feca7aa0157d64f8ad8cf0b3172f9c90026d97694d2ef7ffd9a8ef/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/dec1ab20828babed56cfd0b08d5347742a667c9223a02a14e0954425ebf27844/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/5cd440891a9533efbac9ce20b2477794a62729aa0d713ab17e9f95f6fab510f5/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/633a37591199412fb08f4d9eee9b259642744d1d6242dcdc0562cdc906686beb/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/304eea2fc464c2403000df725664970698f2b1d73ff6f1568bfd2cba1fc63bf3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/407a9b5a5e5cc6fc8727aebafa1919cf20b1882a9efa1af8dc1fa60dd082b549/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/75c0abe1b92d186d6e59c5c7a28773adacdffbb5498e8ed6155e1fa1af5fd64b/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/72bdf079218f6978f1712a1522a2d34b1fd7575603116a7df51d6f349bdd8c6b/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/eea93851742a70d19647d383a6f6eaf67bdea525a681ec0f8b114c5246a4da23/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/1cf3e61cc261318e0260a9065d98febf69ddb24e65d644224a865b2ec2ab0c7d/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/9eb08bed6f84ec62dd1d151fecbd88f4137ef61e4bde325518cb6941e4cf519d/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/e12277550a4845b54bbfda5d8d4b87e619336df247da87c3fe65066ed9d9dba3/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/bc21f983ae6bbccc7109bc2f563258ff2a0adb1f400ddd3a499b43280481a215/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/1885565f79f1ff7b0bbf1a3cb50e7747ca1156a002b9d4b346f1373e1d4e9f60/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/20d168d9360690353bb7c77e8ef873cb0c5c15bcd67cc5ae2a920d2a17f9807b/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/1f96e8f067fd407d72fd7cf94dc76268dc5903f89b61c02c4dc6f0a4abd1a486/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/395f326045475620250b61e126bcc28e66ec240f23c2c2bd13e25759ca657feb/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/3fde11ce5b7d4ae7cdf2d7dc35273b12598a76646b88fb9c5bf67a94c29fc8f4/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/8b17e854c5a6c67b164f588060a3eda4562c106f8da5a37a8343510e08a05425/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/2cfb261455928e89b3eb36fd4abb94b43df4910cf17973671bc8c675a760de84/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/8c723ab3687250bc80a678e59a7644ef7a7bae81c5bdf9a75c9756d995e789cf/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/2f9c71d5d8f7eba56afd49d7c9619ff78dce839d7111537465b71ba9656c3196/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/a8b2aa42933d65b34115e1fed80025ea766de9f47a0049d25137b8ab960444ff/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/4dcd3446ab8f1805d2f113ec0f9dc3e5a2e125fe3f339eb20df3114c20105068/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4ac9d67187d067b74707dc25507b5ac492978342b369d87326dfe867aa2a9dc7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/60e64baa81c522705959413b6118a7946add8c951ebfd1e05fbe2515902995b7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/923f66bdd08bbdf70c8fbcb5f7ffd22c94c029da97690b44734542e2aa03f553/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/0fe7eae2e08566dbc95e8e637152e5d7c250e9fe6911ad6f596b62aa8be637e6/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/74094deec4a597eff792885afddff87928a24eac7e72fb24b766cc30dcb7e5fe/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6ef1c9b11fe4f530379a2b0d1ec591fb60df1de1b027c3b487f9b9ecb14ad5e1/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4c6bfb3bdb7724c5b0209c592ac563cfa9240c6f12ecac3fb0db291633df2663/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/0009fd4117ae865a561c64cdadbd07eb282e455aa19f7115451684a88a86118a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b19f6dac8d89fc4fd3c6d4023ab448371c4323c4759ff30227c087989efa0a0f/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.011721 5473 manager.go:217] Machine: {Timestamp:2026-11-01 14:17:41.011261777 +0000 UTC m=+0.069413619 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100070 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24883c1eda0f9dcd42bce059f67a2e SystemUUID:ec24883c-1eda-0f9d-cd42-bce059f67a2e BootID:d151a1b6-cc4b-4b19-a382-f841eb4cfa55 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5373225-4d3f-451e-ab69-45fee9340973/volumes/kubernetes.io~projected/kube-api-access-54vgg DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~projected/kube-api-access-pqzgz DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volumes/kubernetes.io~projected/kube-api-access-xbqx8 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3880723f-671b-4e04-bfa6-132e59dd9145/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:357 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:362 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~projected/kube-api-access-5c54q DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/885c0b8244be00b3ef13ab616330d7bb6ac615e5f195fc948dd555a85099f2b6/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61db8c12-1910-483d-9ad4-8c491354c549/volumes/kubernetes.io~projected/kube-api-access-r7g56 DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35cb9899-1884-4d74-8951-1a1c4a56b9e9/volumes/kubernetes.io~projected/kube-api-access-mx4cj DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:360 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:363 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8a11aea-1720-432f-a0e4-63a9acbb1efa/volumes/kubernetes.io~projected/kube-api-access-fj9vt DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~projected/kube-api-access-p9pdt DeviceMajor:0 DeviceMinor:365 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:363 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8016d8ca3b2656042672ac1244db0119e114605ee83cd9f8c1756bcf0106b998/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8a11aea-1720-432f-a0e4-63a9acbb1efa/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:340 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/242d8593249c4c8c218aee6880e24551db8cd413584e8f81770c8dcc6e94c299/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b241b884-12de-43ee-af05-7a92bff07b42/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/700ebc96-8068-4533-9376-52fe515cbc3e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3880723f-671b-4e04-bfa6-132e59dd9145/volumes/kubernetes.io~projected/kube-api-access-xtvvx DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:361 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/80966b44-0240-48ab-8528-3ebcfa9c2c12/volumes/kubernetes.io~projected/kube-api-access-xnj8k DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93f8a63fbed9a357723b8879a7c3971572685c32e6c0800c749be26b35a1acda/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a645a5fa31fc5cbdf060837bd1aefe8bee0f4313132efd32e680d3f12c1dc16a/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad629fa2f0e11e0ba3b18239fab16f8b1622cb52b6ef3dea16b1301a7515c327/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:278 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d/volumes/kubernetes.io~projected/kube-api-access-wjp2v DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:364 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1366220d852e1de9f9aa6eaae85302f283aa6af40b150effa8dd9950cdcb37c7/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7bd898ec05073016eae0b942a4f933adffdc11e82fe9e4af7df1748c2d23721/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:275 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/281050d7-54dd-4003-abe4-aa99c6c68a3a/volumes/kubernetes.io~projected/kube-api-access-rczrt DeviceMajor:0 DeviceMinor:341 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/13ea7b9e509a7402b781217f61307fc462872826f9ad1670ff3a3e0b727f3368/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~projected/kube-api-access-pd2z4 DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:363 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/717175a0-02eb-40d7-b696-828a3cb7fcaa/volumes/kubernetes.io~projected/kube-api-access-clvll DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8987e05c983c22397b4f0f5c92cb965f1c7d5d1628f56a17819b7f1bc09dedbc/userdata/shm DeviceMajor:0 DeviceMinor:271 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:281 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03b6d85d-bd06-4e6c-9b9b-93fc59812d24/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44f79f5dd744c62068dbbc091956730d036135c7777fdce6ea6cd2a6a096112d/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dabe07b7-a91c-42ef-aa55-2d2783da8093/volumes/kubernetes.io~projected/kube-api-access-gsqpb DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ad903cc-65c9-4834-88c2-1ada8e4ae38c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d9214a63e9d78bf4be8a9f9630f8e685699f92724f29b1ffc38a5282f746011/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5deaba9bbac383e0bc9a12f13bd7b29650d5a910da993dcbff8d798422e79b2/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:363 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31e7979ccd9c75d3682a055fe12f415bc37ef03585b302e3b73e261d36c97922/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:276 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25cd7543c906ad462da376861dfba4373e0b74522cce2d09f25201b2b3002883/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9d5df786-479c-485b-ba4e-255d1cc48cd5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1ad903cc-65c9-4834-88c2-1ada8e4ae38c/volumes/kubernetes.io~projected/kube-api-access-ddc46 DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/12989f4a-35d4-4dd5-929a-add087567e74/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9d5df786-479c-485b-ba4e-255d1cc48cd5/volumes/kubernetes.io~projected/kube-api-access-xxnln DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9dac2f57ff5c061d749a8b1cc387c3e0b65fff68da234d8da0e750d721d35ae/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:280 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/99114e06-1d8b-4e06-aa86-4436de92e794/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d915508c67121568c2fecd45a92e82dfbb3aebdd4237b8ed6b7ffacc71245d55/userdata/shm DeviceMajor:0 DeviceMinor:345 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:358 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b6a34a13-3327-4384-bc2d-09e30659af46/volumes/kubernetes.io~projected/kube-api-access-4rqg5 DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:242d8593249c4c8 MacAddress:ce:dd:b6:42:c4:a9 Speed:10000 Mtu:8901} {Name:31e7979ccd9c75d MacAddress:4e:e6:f0:89:d3:2f Speed:10000 Mtu:8901} {Name:44f79f5dd744c62 MacAddress:32:14:5d:1c:e7:ea Speed:10000 Mtu:8901} {Name:501807be039b39b MacAddress:0a:4e:1e:35:1a:75 Speed:10000 Mtu:8901} {Name:5d9214a63e9d78b MacAddress:92:cd:dc:b6:ab:9d Speed:10000 Mtu:8901} {Name:9d93d0e5e13587c MacAddress:e6:b1:fd:68:36:9e Speed:10000 Mtu:8901} {Name:a645a5fa31fc5cb MacAddress:66:2f:f4:a2:85:93 Speed:10000 Mtu:8901} {Name:b5deaba9bbac383 MacAddress:8a:ba:5c:71:24:fa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c8:d6:b5:b7 Speed:0 Mtu:9001} {Name:br-int MacAddress:0a:eb:fd:61:86:2d Speed:0 Mtu:8901} {Name:d915508c6712156 MacAddress:3e:5e:35:6d:0d:17 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c8:d6:b5:b7 Speed:0 Mtu:9001} {Name:f9dac2f57ff5c06 MacAddress:ce:11:b6:99:3e:64 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:e9:10:9a:fb:26 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:3c:b0:1a:31:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012104 5473 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012228 5473 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012508 5473 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012525 5473 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-109.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012662 5473 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012670 5473 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012681 5473 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012699 5473 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012773 5473 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.012928 5473 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013048 5473 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013081 5473 kubelet.go:493] "Attempting to sync node with API server" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013095 5473 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013105 5473 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013113 5473 kubelet.go:405] "Adding apiserver pod source" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013123 5473 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013808 5473 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.013824 5473 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 14:17:41.016019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.014475 5473 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 14:17:41.030152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030130 5473 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 14:17:41.030152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030148 5473 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 14:17:41.030317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030265 5473 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 14:17:41.030495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030483 5473 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 14:17:41.030528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030513 5473 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 14:17:41.030620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030610 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030625 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030632 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030638 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030643 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030649 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030656 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030662 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030669 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 14:17:41.030670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030675 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 14:17:41.031044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030690 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 14:17:41.031044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030698 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 14:17:41.031044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030721 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 14:17:41.031044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.030727 5473 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 14:17:41.034104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.034088 5473 server.go:1263] "Started kubelet" Nov 01 14:17:41.034232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.034189 5473 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 14:17:41.034642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.034309 5473 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 14:17:41.034642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.034400 5473 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 14:17:41.034795 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.034705 5473 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 14:17:41.034700 ip-10-0-131-109 systemd[1]: Started Kubernetes Kubelet. Nov 01 14:17:41.035401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.035380 5473 server.go:317] "Adding debug handlers to kubelet server" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.037838 5473 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.038033 5473 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.038087 5473 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 14:11:29 +0000 UTC" deadline="2028-04-30 01:59:44.687395649 +0000 UTC" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.038116 5473 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13091h42m3.649282586s" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.038149 5473 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.038181 5473 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.038188 5473 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 14:17:41.039522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.038499 5473 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 14:17:41.040980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.040962 5473 factory.go:55] Registering systemd factory Nov 01 14:17:41.041060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.040983 5473 factory.go:223] Registration of the systemd container factory successfully Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041780 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12989f4a-35d4-4dd5-929a-add087567e74" volumeName="kubernetes.io/projected/12989f4a-35d4-4dd5-929a-add087567e74-kube-api-access-xbqx8" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041882 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad903cc-65c9-4834-88c2-1ada8e4ae38c" volumeName="kubernetes.io/empty-dir/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-data-volume" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041920 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="281050d7-54dd-4003-abe4-aa99c6c68a3a" volumeName="kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041944 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880723f-671b-4e04-bfa6-132e59dd9145" volumeName="kubernetes.io/secret/3880723f-671b-4e04-bfa6-132e59dd9145-metrics-tls" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041958 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041972 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.041990 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" volumeName="kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042011 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" volumeName="kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042057 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" volumeName="kubernetes.io/empty-dir/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-textfile" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042074 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12989f4a-35d4-4dd5-929a-add087567e74" volumeName="kubernetes.io/secret/12989f4a-35d4-4dd5-929a-add087567e74-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042093 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717175a0-02eb-40d7-b696-828a3cb7fcaa" volumeName="kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-daemon-config" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042107 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-tls" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042121 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042139 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042159 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" volumeName="kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042177 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" volumeName="kubernetes.io/projected/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-kube-api-access-wjp2v" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042191 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad903cc-65c9-4834-88c2-1ada8e4ae38c" volumeName="kubernetes.io/secret/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042209 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880723f-671b-4e04-bfa6-132e59dd9145" volumeName="kubernetes.io/configmap/3880723f-671b-4e04-bfa6-132e59dd9145-config-volume" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042237 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61db8c12-1910-483d-9ad4-8c491354c549" volumeName="kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-tls" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042251 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700ebc96-8068-4533-9376-52fe515cbc3e" volumeName="kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-tmp" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042269 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042280 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042298 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12989f4a-35d4-4dd5-929a-add087567e74" volumeName="kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-env-overrides" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042311 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35cb9899-1884-4d74-8951-1a1c4a56b9e9" volumeName="kubernetes.io/projected/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kube-api-access-mx4cj" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042323 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880723f-671b-4e04-bfa6-132e59dd9145" volumeName="kubernetes.io/empty-dir/3880723f-671b-4e04-bfa6-132e59dd9145-tmp-dir" seLinuxMountContext="" Nov 01 14:17:41.042814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042340 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61db8c12-1910-483d-9ad4-8c491354c549" volumeName="kubernetes.io/configmap/61db8c12-1910-483d-9ad4-8c491354c549-metrics-client-ca" seLinuxMountContext="" Nov 01 14:17:41.046768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.043000 5473 factory.go:153] Registering CRI-O factory Nov 01 14:17:41.046768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.043023 5473 factory.go:223] Registration of the crio container factory successfully Nov 01 14:17:41.046768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.043081 5473 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 14:17:41.046768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.043101 5473 factory.go:103] Registering Raw factory Nov 01 14:17:41.046768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.043121 5473 manager.go:1199] Started watching for new ooms in manager Nov 01 14:17:41.046768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.043532 5473 manager.go:319] Starting recovery of all containers Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.042354 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047656 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b241b884-12de-43ee-af05-7a92bff07b42" volumeName="kubernetes.io/secret/b241b884-12de-43ee-af05-7a92bff07b42-agent-certs" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047718 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047742 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047775 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" volumeName="kubernetes.io/projected/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-kube-api-access-5c54q" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047793 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047826 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047851 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dabe07b7-a91c-42ef-aa55-2d2783da8093" volumeName="kubernetes.io/configmap/dabe07b7-a91c-42ef-aa55-2d2783da8093-iptables-alerter-script" seLinuxMountContext="" Nov 01 14:17:41.047922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047878 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5373225-4d3f-451e-ab69-45fee9340973" volumeName="kubernetes.io/empty-dir/f5373225-4d3f-451e-ab69-45fee9340973-tmp-dir" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.047980 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048037 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5df786-479c-485b-ba4e-255d1cc48cd5" volumeName="kubernetes.io/secret/9d5df786-479c-485b-ba4e-255d1cc48cd5-cert" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048059 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048074 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048098 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61db8c12-1910-483d-9ad4-8c491354c549" volumeName="kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048117 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048136 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048162 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80966b44-0240-48ab-8528-3ebcfa9c2c12" volumeName="kubernetes.io/projected/80966b44-0240-48ab-8528-3ebcfa9c2c12-kube-api-access-xnj8k" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048180 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048202 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dabe07b7-a91c-42ef-aa55-2d2783da8093" volumeName="kubernetes.io/projected/dabe07b7-a91c-42ef-aa55-2d2783da8093-kube-api-access-gsqpb" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048243 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8a11aea-1720-432f-a0e4-63a9acbb1efa" volumeName="kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048273 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" volumeName="kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-binary-copy" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048301 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61db8c12-1910-483d-9ad4-8c491354c549" volumeName="kubernetes.io/projected/61db8c12-1910-483d-9ad4-8c491354c549-kube-api-access-r7g56" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048317 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048341 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d5df786-479c-485b-ba4e-255d1cc48cd5" volumeName="kubernetes.io/projected/9d5df786-479c-485b-ba4e-255d1cc48cd5-kube-api-access-xxnln" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048357 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b241b884-12de-43ee-af05-7a92bff07b42" volumeName="kubernetes.io/configmap/b241b884-12de-43ee-af05-7a92bff07b42-konnectivity-ca" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048371 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-metrics-server-audit-profiles" seLinuxMountContext="" Nov 01 14:17:41.048383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048393 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/empty-dir/b6a34a13-3327-4384-bc2d-09e30659af46-audit-log" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048408 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" volumeName="kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-metrics-client-ca" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048425 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048440 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048480 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048498 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700ebc96-8068-4533-9376-52fe515cbc3e" volumeName="kubernetes.io/projected/700ebc96-8068-4533-9376-52fe515cbc3e-kube-api-access-pqzgz" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048511 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12989f4a-35d4-4dd5-929a-add087567e74" volumeName="kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-config" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048527 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad903cc-65c9-4834-88c2-1ada8e4ae38c" volumeName="kubernetes.io/projected/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-api-access-ddc46" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048545 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/projected/b6a34a13-3327-4384-bc2d-09e30659af46-kube-api-access-4rqg5" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048560 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6a34a13-3327-4384-bc2d-09e30659af46" volumeName="kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-client-ca-bundle" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048583 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048597 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="700ebc96-8068-4533-9376-52fe515cbc3e" volumeName="kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-etc-tuned" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048612 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80966b44-0240-48ab-8528-3ebcfa9c2c12" volumeName="kubernetes.io/configmap/80966b44-0240-48ab-8528-3ebcfa9c2c12-serviceca" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048630 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048642 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048660 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" volumeName="kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-tls" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048674 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3880723f-671b-4e04-bfa6-132e59dd9145" volumeName="kubernetes.io/projected/3880723f-671b-4e04-bfa6-132e59dd9145-kube-api-access-xtvvx" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048687 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048706 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048723 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048742 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8a11aea-1720-432f-a0e4-63a9acbb1efa" volumeName="kubernetes.io/projected/e8a11aea-1720-432f-a0e4-63a9acbb1efa-kube-api-access-fj9vt" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048760 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" volumeName="kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048781 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ad903cc-65c9-4834-88c2-1ada8e4ae38c" volumeName="kubernetes.io/configmap/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048800 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" volumeName="kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048819 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048844 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5373225-4d3f-451e-ab69-45fee9340973" volumeName="kubernetes.io/projected/f5373225-4d3f-451e-ab69-45fee9340973-kube-api-access-54vgg" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048861 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12989f4a-35d4-4dd5-929a-add087567e74" volumeName="kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-script-lib" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048884 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99114e06-1d8b-4e06-aa86-4436de92e794" volumeName="kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048902 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048918 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048942 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048958 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048978 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717175a0-02eb-40d7-b696-828a3cb7fcaa" volumeName="kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-cni-binary-copy" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.048994 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="717175a0-02eb-40d7-b696-828a3cb7fcaa" volumeName="kubernetes.io/projected/717175a0-02eb-40d7-b696-828a3cb7fcaa-kube-api-access-clvll" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.049013 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.049034 5473 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8121105-7c56-42bf-89a2-858453626862" volumeName="kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.049612 5473 reconstruct.go:97] "Volume reconstruction finished" Nov 01 14:17:41.050150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.049622 5473 reconciler.go:29] "Reconciler: start to sync state" Nov 01 14:17:41.072059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.072020 5473 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 14:17:41.074040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.073718 5473 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 14:17:41.074040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.073751 5473 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 14:17:41.074040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.073780 5473 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 14:17:41.076020 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.074129 5473 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 14:17:41.085473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.085447 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" exitCode=0 Nov 01 14:17:41.093009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.092987 5473 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="dd7a29076bbca7d8c7cc135dd652913784404cc7610bb527ae917936b82f0d22" exitCode=0 Nov 01 14:17:41.093009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.093009 5473 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="d114e7466a84788d53b975085bddf939578542b161fd3795eae15078777c668d" exitCode=0 Nov 01 14:17:41.093009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.093018 5473 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="e47432d018708189c273a8109d3085793b23cc663d971c8ce21359cc1e9bcf6a" exitCode=0 Nov 01 14:17:41.093204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.093027 5473 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="2ad02c6faa02eb0b9d0853d25b9d01646e20c5aa51c059877c5938cc5e35aa1b" exitCode=0 Nov 01 14:17:41.093204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.093035 5473 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="a4d1e092540830825b1016ed4ba0f103439773957506a1124126f3ff2e5ba0a2" exitCode=0 Nov 01 14:17:41.093204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.093042 5473 generic.go:356] "Generic (PLEG): container finished" podID="b25b5ead-9a4d-489b-a628-a9fa2eca2b2d" containerID="924a4d08bca526bd8f0e1efd5f8ed40293e0239c0b9a86871318e53d95a55631" exitCode=0 Nov 01 14:17:41.114994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.114920 5473 generic.go:356] "Generic (PLEG): container finished" podID="a71522b17879685585fdaa072a2e7d0d" containerID="bd3bb9834634f3d4c6127f08f9ce422e30668232b9a3238a573ef90061811ca4" exitCode=0 Nov 01 14:17:41.119581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.119556 5473 generic.go:356] "Generic (PLEG): container finished" podID="03b6d85d-bd06-4e6c-9b9b-93fc59812d24" containerID="9b87fb4a771e40294e209e145decebe9daeb366bbb2ad2e591cbb637477dc0bc" exitCode=0 Nov 01 14:17:41.128773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.128744 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" exitCode=0 Nov 01 14:17:41.138272 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.138242 5473 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:17:41.146675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.146656 5473 manager.go:324] Recovery completed Nov 01 14:17:41.167548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.167523 5473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:17:41.169799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.169781 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:17:41.169907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.169811 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:17:41.169907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.169822 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:17:41.172292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172275 5473 cpu_manager.go:226] "Starting" policy="none" Nov 01 14:17:41.172292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172289 5473 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 14:17:41.172384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172300 5473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:17:41.172420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172307 5473 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 14:17:41.172523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172510 5473 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 14:17:41.172556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172523 5473 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 14:17:41.172595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172587 5473 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 14:17:41.172625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172595 5473 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 14:17:41.172625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172600 5473 policy_none.go:50] "Start" Nov 01 14:17:41.172625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172609 5473 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 14:17:41.172625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172618 5473 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 14:17:41.172747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172737 5473 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 14:17:41.172776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172747 5473 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 14:17:41.172776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.172753 5473 policy_none.go:44] "Start" Nov 01 14:17:41.173411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.173392 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:17:41.173484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.173429 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:17:41.173484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.173441 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:17:41.174358 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.174341 5473 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 14:17:41.177386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.177357 5473 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 14:17:41.179955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.179942 5473 manager.go:341] "Starting Device Plugin manager" Nov 01 14:17:41.180039 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.179967 5473 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 14:17:41.180039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.179976 5473 server.go:87] "Starting device plugin registration server" Nov 01 14:17:41.180254 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180241 5473 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 14:17:41.180296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180256 5473 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 14:17:41.180365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180352 5473 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 14:17:41.180489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180451 5473 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 14:17:41.180489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180497 5473 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 14:17:41.180664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180504 5473 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 14:17:41.182564 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.182538 5473 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 14:17:41.182672 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.182611 5473 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-109.us-west-2.compute.internal\" not found" Nov 01 14:17:41.182828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.182811 5473 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 14:17:41.182901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.182851 5473 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 14:17:41.184553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.180549 5473 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T14:17:41.180482758Z","UUID":"6f323bdb-816a-451d-994c-de8b74c05ef4","Handler":null,"Name":"","Endpoint":""} Nov 01 14:17:41.206226 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.206194 5473 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:17:41.206348 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.206253 5473 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 14:17:41.251496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.251468 5473 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:17:41.251496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.251494 5473 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:17:41.280852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.280826 5473 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 14:17:41.281857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.281840 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 14:17:41.281932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.281871 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 14:17:41.281932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.281881 5473 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-109.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 01 14:17:41.282001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.281934 5473 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.297732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.297705 5473 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 14:17:41.374654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.374574 5473 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal"] Nov 01 14:17:41.375197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375162 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"fbed3511ed9fe6dd871e46aab86e24ee","Type":"ContainerStarted","Data":"42b8554dffb0abc8b66679ae253f9f9019c98e7e6a04019e2fa616d4446442e6"} Nov 01 14:17:41.375265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375205 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"fbed3511ed9fe6dd871e46aab86e24ee","Type":"ContainerStarted","Data":"55453fd6ca0a995ee85952362affa4accbd500fcdfdbb703b7cce4ba879556e1"} Nov 01 14:17:41.375265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375231 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"a71522b17879685585fdaa072a2e7d0d","Type":"ContainerStarted","Data":"5aef96539c7b00e8ee67f1736b2ce58bd5acb585f0dc8152c999197f9ad77aed"} Nov 01 14:17:41.375265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375240 5473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.375385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375245 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"a71522b17879685585fdaa072a2e7d0d","Type":"ContainerDied","Data":"bd3bb9834634f3d4c6127f08f9ce422e30668232b9a3238a573ef90061811ca4"} Nov 01 14:17:41.375385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375339 5473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.375385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.375351 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" event={"ID":"a71522b17879685585fdaa072a2e7d0d","Type":"ContainerStarted","Data":"49bd22b55d11e0c38f929e27330854fea534655b8771f190858e63b11eefdb17"} Nov 01 14:17:41.383575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.383558 5473 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.383671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.383618 5473 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.437284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.437261 5473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 14:17:41.437406 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.437308 5473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.437796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.437782 5473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 14:17:41.437835 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:41.437819 5473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.507914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.507889 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbed3511ed9fe6dd871e46aab86e24ee-config\") pod \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"fbed3511ed9fe6dd871e46aab86e24ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.507914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.507920 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.508111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.507938 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.608635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.608598 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbed3511ed9fe6dd871e46aab86e24ee-config\") pod \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"fbed3511ed9fe6dd871e46aab86e24ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.608635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.608634 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.608819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.608660 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.608819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.608712 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.608819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.608725 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a71522b17879685585fdaa072a2e7d0d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"a71522b17879685585fdaa072a2e7d0d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:41.608819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:41.608725 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbed3511ed9fe6dd871e46aab86e24ee-config\") pod \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" (UID: \"fbed3511ed9fe6dd871e46aab86e24ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:42.014315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.014283 5473 apiserver.go:52] "Watching apiserver" Nov 01 14:17:42.109716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.109665 5473 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 14:17:42.110324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.110296 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-jbjh4","openshift-ovn-kubernetes/ovnkube-node-9vsn2","openshift-cluster-node-tuning-operator/tuned-ch8fm","openshift-dns/dns-default-fx4jc","openshift-monitoring/metrics-server-674fc8bc5b-hm9rs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl","openshift-dns/node-resolver-42stp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-zx5dd","openshift-multus/multus-additional-cni-plugins-cmjxb","openshift-network-diagnostics/network-check-target-t7sh8","openshift-network-operator/iptables-alerter-528rj","kube-system/konnectivity-agent-82phh","openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-48mxn","openshift-insights/insights-runtime-extractor-4279c","openshift-monitoring/node-exporter-pg8hh","kube-system/global-pull-secret-syncer-ckfsz","kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-f72xc"] Nov 01 14:17:42.133496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.133472 5473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:42.133617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.133602 5473 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:42.140650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.140628 5473 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 14:17:42.160123 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160085 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.160396 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160130 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.160536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160453 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.160643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160541 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.160643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160551 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.160643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160585 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.160643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160589 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.160643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160591 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 14:17:42.160643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160636 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 14:17:42.160930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160675 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x2nj9\"" Nov 01 14:17:42.160930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160696 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 14:17:42.160930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160784 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m46fn\"" Nov 01 14:17:42.160930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160847 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mpq57\"" Nov 01 14:17:42.161066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.160980 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-smpqh\"" Nov 01 14:17:42.161066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.161032 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ftvdm\"" Nov 01 14:17:42.161152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.161079 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 14:17:42.161204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.161148 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.163272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.163259 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.211059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211032 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-host\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211069 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.211204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211089 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysconfig\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211106 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.211204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211172 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.211406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211305 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:42.211406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211333 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-tmp\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211312 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.211406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211358 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-var-lib-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.211406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211377 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.211406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211395 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.211605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211456 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-tmp\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211516 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-client-certs\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.211605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211588 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-cnibin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.211691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211632 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.211691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211672 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.211747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211705 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-socket-dir-parent\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.211747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211721 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-systemd\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211737 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-run\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211763 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3880723f-671b-4e04-bfa6-132e59dd9145-metrics-tls\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.211880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211801 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-conf\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211837 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.211880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211868 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-env-overrides\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.211998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211886 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.211998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211948 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/700ebc96-8068-4533-9376-52fe515cbc3e-kube-api-access-pqzgz\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.211998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.211970 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.212091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212051 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b241b884-12de-43ee-af05-7a92bff07b42-konnectivity-ca\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:17:42.212185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212162 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.212232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212196 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dabe07b7-a91c-42ef-aa55-2d2783da8093-iptables-alerter-script\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.212269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212232 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-lib-modules\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.212269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212254 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.212342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212274 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.212342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212289 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-dbus\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.212342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212307 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.212448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212330 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-log-socket\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.212448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212381 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbqx8\" (UniqueName: \"kubernetes.io/projected/12989f4a-35d4-4dd5-929a-add087567e74-kube-api-access-xbqx8\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.212448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212410 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c54q\" (UniqueName: \"kubernetes.io/projected/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-kube-api-access-5c54q\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.212448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212421 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dabe07b7-a91c-42ef-aa55-2d2783da8093-iptables-alerter-script\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.212598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212461 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.212598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212505 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-metrics-server-audit-profiles\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.212598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212537 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-multus\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.212598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212574 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-daemon-config\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212607 5473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212614 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-kubernetes\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:42.212662 5473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212681 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212700 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212717 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b241b884-12de-43ee-af05-7a92bff07b42-agent-certs\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212735 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212750 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-netns\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.212769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212774 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.213027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212814 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3880723f-671b-4e04-bfa6-132e59dd9145-config-volume\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.213027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212865 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.213027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212905 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-root\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212971 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-textfile\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.212998 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-crio-socket\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213026 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-textfile\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213024 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cnibin\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213053 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213085 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-netd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213112 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61db8c12-1910-483d-9ad4-8c491354c549-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213142 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-data-volume\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.213202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213183 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213211 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-data-volume\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213248 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-config\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213290 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-tls\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213336 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213359 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-node-log\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213376 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-wtmp\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-metrics-client-ca\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213436 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-accelerators-collector-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213520 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213564 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213587 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213598 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-client-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213619 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213628 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-conf-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213651 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/99114e06-1d8b-4e06-aa86-4436de92e794-original-pull-secret\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213656 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxnln\" (UniqueName: \"kubernetes.io/projected/9d5df786-479c-485b-ba4e-255d1cc48cd5-kube-api-access-xxnln\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213698 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a34a13-3327-4384-bc2d-09e30659af46-audit-log\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.213771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213762 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjp2v\" (UniqueName: \"kubernetes.io/projected/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-kube-api-access-wjp2v\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213785 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b6a34a13-3327-4384-bc2d-09e30659af46-audit-log\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213817 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12989f4a-35d4-4dd5-929a-add087567e74-ovn-node-metrics-cert\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213833 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213851 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddc46\" (UniqueName: \"kubernetes.io/projected/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-api-access-ddc46\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213929 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213967 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.213992 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214016 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214042 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-systemd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214078 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214114 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g56\" (UniqueName: \"kubernetes.io/projected/61db8c12-1910-483d-9ad4-8c491354c549-kube-api-access-r7g56\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214141 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-modprobe-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214163 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-var-lib-kubelet\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214232 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-ovn\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214276 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-etc-tuned\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214317 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214350 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214369 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214385 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/700ebc96-8068-4533-9376-52fe515cbc3e-etc-tuned\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214481 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214507 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214536 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214557 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5373225-4d3f-451e-ab69-45fee9340973-hosts-file\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214574 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-multus-certs\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214589 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-slash\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214611 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-bin\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214669 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsqpb\" (UniqueName: \"kubernetes.io/projected/dabe07b7-a91c-42ef-aa55-2d2783da8093-kube-api-access-gsqpb\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214695 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.214710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214710 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80966b44-0240-48ab-8528-3ebcfa9c2c12-serviceca\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214726 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214803 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214836 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvll\" (UniqueName: \"kubernetes.io/projected/717175a0-02eb-40d7-b696-828a3cb7fcaa-kube-api-access-clvll\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214861 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-kubelet\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214885 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-netns\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214918 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.214998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.214967 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215000 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqg5\" (UniqueName: \"kubernetes.io/projected/b6a34a13-3327-4384-bc2d-09e30659af46-kube-api-access-4rqg5\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215031 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4cj\" (UniqueName: \"kubernetes.io/projected/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kube-api-access-mx4cj\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215058 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5df786-479c-485b-ba4e-255d1cc48cd5-cert\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215085 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215113 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215151 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215195 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5373225-4d3f-451e-ab69-45fee9340973-tmp-dir\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215237 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-k8s-cni-cncf-io\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215253 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215276 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnj8k\" (UniqueName: \"kubernetes.io/projected/80966b44-0240-48ab-8528-3ebcfa9c2c12-kube-api-access-xnj8k\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215297 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5373225-4d3f-451e-ab69-45fee9340973-tmp-dir\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215315 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-system-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215351 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880723f-671b-4e04-bfa6-132e59dd9145-tmp-dir\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215398 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-device-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215403 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3880723f-671b-4e04-bfa6-132e59dd9145-tmp-dir\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215453 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-systemd-units\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215473 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215490 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215507 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-tls\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215589 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vgg\" (UniqueName: \"kubernetes.io/projected/f5373225-4d3f-451e-ab69-45fee9340973-kube-api-access-54vgg\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215623 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-kubelet\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215650 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-script-lib\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215701 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.215738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215705 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8a11aea-1720-432f-a0e4-63a9acbb1efa-metrics-certs\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215738 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215785 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80966b44-0240-48ab-8528-3ebcfa9c2c12-host\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215820 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-etc-kubernetes\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215846 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-hostroot\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215868 5473 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215890 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvvx\" (UniqueName: \"kubernetes.io/projected/3880723f-671b-4e04-bfa6-132e59dd9145-kube-api-access-xtvvx\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:17:42.215912 5473 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-109.us-west-2.compute.internal" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215944 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj9vt\" (UniqueName: \"kubernetes.io/projected/e8a11aea-1720-432f-a0e4-63a9acbb1efa-kube-api-access-fj9vt\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.215981 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216025 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216053 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216080 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-bin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.216264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216130 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-sys\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216289 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216334 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-etc-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216360 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-sys\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216384 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dabe07b7-a91c-42ef-aa55-2d2783da8093-host-slash\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216412 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216461 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216493 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-kubelet-config\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216524 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.216701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.216567 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-cni-binary-copy\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.249421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.249392 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:42.259924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.259896 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 14:17:42.260080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.260021 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 14:17:42.260080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.260028 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 14:17:42.260152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.260093 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-9p5x8\"" Nov 01 14:17:42.260849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.260828 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 14:17:42.261053 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.261032 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 14:17:42.261342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.261315 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 14:17:42.261432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.261369 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 14:17:42.261516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.261501 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.261593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.261581 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 14:17:42.261677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.261661 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 14:17:42.262090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262057 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-env-overrides\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.262179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262094 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 14:17:42.262263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262203 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 14:17:42.262433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262378 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 14:17:42.262433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262406 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.262574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262451 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 14:17:42.262574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262542 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.262840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262822 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 14:17:42.262929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262911 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.263002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.262986 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.263311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263297 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-metrics-server-audit-profiles\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.263647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263627 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 14:17:42.263647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263627 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 14:17:42.263804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263784 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvptn\"" Nov 01 14:17:42.263804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263800 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 14:17:42.263949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263824 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 14:17:42.263949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263844 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 14:17:42.263949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263877 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 14:17:42.263949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263880 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.263949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263896 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263960 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.263962 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264027 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264040 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264082 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fnkb\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264122 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264147 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264158 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264162 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 14:17:42.264201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264208 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264271 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264450 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264465 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264495 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264594 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264658 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjzzz\"" Nov 01 14:17:42.264699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264678 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5wqpw\"" Nov 01 14:17:42.265023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264715 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.265023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264715 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.265023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.264926 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.265168 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265137 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265339 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265356 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-24rb9\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265344 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265414 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265429 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265447 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7pw86\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265456 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80966b44-0240-48ab-8528-3ebcfa9c2c12-serviceca\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265488 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d5df786-479c-485b-ba4e-255d1cc48cd5-cert\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265492 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-uu6sl2qqfhq0\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265557 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265564 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265621 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265618 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqzgz\" (UniqueName: \"kubernetes.io/projected/700ebc96-8068-4533-9376-52fe515cbc3e-kube-api-access-pqzgz\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265642 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lp9km\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265683 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265691 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vshv8\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265706 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265703 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265743 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265748 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v4966\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265758 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 14:17:42.265775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.265766 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267116 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267242 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-js46j\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267267 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267275 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-cni-binary-copy\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267280 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267513 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8xv6q\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267627 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267520 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-x2zjc\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267819 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.267970 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268055 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268060 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vd72s\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268122 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268392 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268465 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268517 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.268823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268660 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 14:17:42.270031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268967 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-cj7jb\"" Nov 01 14:17:42.270031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.268977 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 14:17:42.270031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.269255 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-29vjcn8bfe0n4\"" Nov 01 14:17:42.270031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.269601 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczrt\" (UniqueName: \"kubernetes.io/projected/281050d7-54dd-4003-abe4-aa99c6c68a3a-kube-api-access-rczrt\") pod \"network-check-target-t7sh8\" (UID: \"281050d7-54dd-4003-abe4-aa99c6c68a3a\") " pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:42.271667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.271644 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.271763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.271676 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.271842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.271824 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-client-certs\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.272581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272388 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3880723f-671b-4e04-bfa6-132e59dd9145-metrics-tls\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.272581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272393 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.272581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272474 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6a34a13-3327-4384-bc2d-09e30659af46-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.272581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272482 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b241b884-12de-43ee-af05-7a92bff07b42-konnectivity-ca\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:17:42.272581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272537 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.272787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272605 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cni-binary-copy\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.272787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272646 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.272787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272704 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.272871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272810 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-daemon-config\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.272978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.272965 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b241b884-12de-43ee-af05-7a92bff07b42-agent-certs\") pod \"konnectivity-agent-82phh\" (UID: \"b241b884-12de-43ee-af05-7a92bff07b42\") " pod="kube-system/konnectivity-agent-82phh" Nov 01 14:17:42.273116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.273102 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3880723f-671b-4e04-bfa6-132e59dd9145-config-volume\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.273808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.273787 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-accelerators-collector-config\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.273808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.273801 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.273957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.273900 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-tls\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.273957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.273907 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-client-ca-bundle\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.273957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.273916 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-config\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.274110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.274029 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.274110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.274059 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12989f4a-35d4-4dd5-929a-add087567e74-ovn-node-metrics-cert\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.274205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.274148 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.274269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.274237 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.275112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.275096 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.276175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.276159 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.276258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.276183 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61db8c12-1910-483d-9ad4-8c491354c549-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.276258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.276236 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b6a34a13-3327-4384-bc2d-09e30659af46-secret-metrics-server-tls\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.276357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.276296 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.310261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.310232 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 14:17:42.310376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.310293 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.310906 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbqx8\" (UniqueName: \"kubernetes.io/projected/12989f4a-35d4-4dd5-929a-add087567e74-kube-api-access-xbqx8\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311012 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4cj\" (UniqueName: \"kubernetes.io/projected/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kube-api-access-mx4cj\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311133 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g56\" (UniqueName: \"kubernetes.io/projected/61db8c12-1910-483d-9ad4-8c491354c549-kube-api-access-r7g56\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311210 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c54q\" (UniqueName: \"kubernetes.io/projected/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-kube-api-access-5c54q\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311340 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311922 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddc46\" (UniqueName: \"kubernetes.io/projected/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-kube-api-access-ddc46\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311937 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnj8k\" (UniqueName: \"kubernetes.io/projected/80966b44-0240-48ab-8528-3ebcfa9c2c12-kube-api-access-xnj8k\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.311950 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vgg\" (UniqueName: \"kubernetes.io/projected/f5373225-4d3f-451e-ab69-45fee9340973-kube-api-access-54vgg\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312124 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312137 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqg5\" (UniqueName: \"kubernetes.io/projected/b6a34a13-3327-4384-bc2d-09e30659af46-kube-api-access-4rqg5\") pod \"metrics-server-674fc8bc5b-hm9rs\" (UID: \"b6a34a13-3327-4384-bc2d-09e30659af46\") " pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312187 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312337 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312506 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsqpb\" (UniqueName: \"kubernetes.io/projected/dabe07b7-a91c-42ef-aa55-2d2783da8093-kube-api-access-gsqpb\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312527 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312628 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvll\" (UniqueName: \"kubernetes.io/projected/717175a0-02eb-40d7-b696-828a3cb7fcaa-kube-api-access-clvll\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.312744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312691 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvvx\" (UniqueName: \"kubernetes.io/projected/3880723f-671b-4e04-bfa6-132e59dd9145-kube-api-access-xtvvx\") pod \"dns-default-fx4jc\" (UID: \"3880723f-671b-4e04-bfa6-132e59dd9145\") " pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:42.313265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312756 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.313265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312788 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxnln\" (UniqueName: \"kubernetes.io/projected/9d5df786-479c-485b-ba4e-255d1cc48cd5-kube-api-access-xxnln\") pod \"ingress-canary-zx5dd\" (UID: \"9d5df786-479c-485b-ba4e-255d1cc48cd5\") " pod="openshift-ingress-canary/ingress-canary-zx5dd" Nov 01 14:17:42.313265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.312865 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjp2v\" (UniqueName: \"kubernetes.io/projected/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-kube-api-access-wjp2v\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.313265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.313047 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj9vt\" (UniqueName: \"kubernetes.io/projected/e8a11aea-1720-432f-a0e4-63a9acbb1efa-kube-api-access-fj9vt\") pod \"network-metrics-daemon-f72xc\" (UID: \"e8a11aea-1720-432f-a0e4-63a9acbb1efa\") " pod="openshift-multus/network-metrics-daemon-f72xc" Nov 01 14:17:42.313773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.313758 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61db8c12-1910-483d-9ad4-8c491354c549-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-j9hdh\" (UID: \"61db8c12-1910-483d-9ad4-8c491354c549\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-j9hdh" Nov 01 14:17:42.313872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.313859 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-metrics-client-ca\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.315020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.315003 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.316149 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.316126 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12989f4a-35d4-4dd5-929a-add087567e74-ovnkube-script-lib\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.316484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.316464 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:17:42.316692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.316674 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.317028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317011 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-bin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.317059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317047 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-sys\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317074 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-etc-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.317153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317102 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-sys\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.317153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317107 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-bin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.317153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317126 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dabe07b7-a91c-42ef-aa55-2d2783da8093-host-slash\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.317153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317130 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-sys\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317156 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-kubelet-config\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317163 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-etc-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317164 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-sys\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317157 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dabe07b7-a91c-42ef-aa55-2d2783da8093-host-slash\") pod \"iptables-alerter-528rj\" (UID: \"dabe07b7-a91c-42ef-aa55-2d2783da8093\") " pod="openshift-network-operator/iptables-alerter-528rj" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317192 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-kubelet-config\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317179 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317238 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-registration-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317253 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-host\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317282 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysconfig\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317314 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-var-lib-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317345 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-var-lib-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317342 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-host\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317361 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysconfig\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317349 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-cnibin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317405 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317434 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-socket-dir-parent\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317453 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317455 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-cnibin\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317458 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-systemd\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317496 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-run\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317513 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-socket-dir-parent\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317522 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-conf\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317511 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-systemd\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317554 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317558 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-run\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.317575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317580 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-lib-modules\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317601 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-dbus\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317620 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-log-socket\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317635 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-sys-fs\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317642 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-multus\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317665 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-kubernetes\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317668 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-log-socket\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317678 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/99114e06-1d8b-4e06-aa86-4436de92e794-dbus\") pod \"global-pull-secret-syncer-ckfsz\" (UID: \"99114e06-1d8b-4e06-aa86-4436de92e794\") " pod="kube-system/global-pull-secret-syncer-ckfsz" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317683 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-conf\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317690 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317720 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-kubernetes\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-netns\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317704 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-cni-multus\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317745 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-socket-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317755 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-root\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317775 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-netns\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317781 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-crio-socket\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317745 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-lib-modules\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317807 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cnibin\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317814 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1ad903cc-65c9-4834-88c2-1ada8e4ae38c-crio-socket\") pod \"insights-runtime-extractor-4279c\" (UID: \"1ad903cc-65c9-4834-88c2-1ada8e4ae38c\") " pod="openshift-insights/insights-runtime-extractor-4279c" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317808 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-root\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317838 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317843 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-cnibin\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317855 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-netd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317881 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317890 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-netd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317903 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-node-log\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317931 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-wtmp\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317959 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-conf-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317972 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-node-log\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317985 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/03b6d85d-bd06-4e6c-9b9b-93fc59812d24-node-exporter-wtmp\") pod \"node-exporter-pg8hh\" (UID: \"03b6d85d-bd06-4e6c-9b9b-93fc59812d24\") " pod="openshift-monitoring/node-exporter-pg8hh" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.317991 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318010 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-conf-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318016 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-systemd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318042 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318060 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-systemd\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318046 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-multus-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318067 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318070 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-modprobe-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318094 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-var-lib-kubelet\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318118 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-ovn\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318137 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-modprobe-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318170 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-var-lib-kubelet\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318169 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318188 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-ovn\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318195 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5373225-4d3f-451e-ab69-45fee9340973-hosts-file\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318242 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-etc-selinux\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318257 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f5373225-4d3f-451e-ab69-45fee9340973-hosts-file\") pod \"node-resolver-42stp\" (UID: \"f5373225-4d3f-451e-ab69-45fee9340973\") " pod="openshift-dns/node-resolver-42stp" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318265 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-multus-certs\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318247 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-multus-certs\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.318263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318295 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-slash\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318317 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-bin\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318346 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-kubelet\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318385 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-slash\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318393 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-cni-bin\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318385 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-netns\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318416 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318417 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-kubelet\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318436 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318456 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-run-openvswitch\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318434 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-host-run-netns\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318452 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318488 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-system-cni-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318497 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b25b5ead-9a4d-489b-a628-a9fa2eca2b2d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cmjxb\" (UID: \"b25b5ead-9a4d-489b-a628-a9fa2eca2b2d\") " pod="openshift-multus/multus-additional-cni-plugins-cmjxb" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318506 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-k8s-cni-cncf-io\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318530 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318553 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-run-k8s-cni-cncf-io\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318554 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-system-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318584 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-device-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318602 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-system-cni-dir\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318605 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-systemd-units\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318622 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12989f4a-35d4-4dd5-929a-add087567e74-systemd-units\") pod \"ovnkube-node-9vsn2\" (UID: \"12989f4a-35d4-4dd5-929a-add087567e74\") " pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318638 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/700ebc96-8068-4533-9376-52fe515cbc3e-etc-sysctl-d\") pod \"tuned-ch8fm\" (UID: \"700ebc96-8068-4533-9376-52fe515cbc3e\") " pod="openshift-cluster-node-tuning-operator/tuned-ch8fm" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318645 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-kubelet\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318655 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35cb9899-1884-4d74-8951-1a1c4a56b9e9-device-dir\") pod \"aws-ebs-csi-driver-node-zb4xl\" (UID: \"35cb9899-1884-4d74-8951-1a1c4a56b9e9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zb4xl" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318681 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-host-var-lib-kubelet\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318671 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80966b44-0240-48ab-8528-3ebcfa9c2c12-host\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318706 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-etc-kubernetes\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318720 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80966b44-0240-48ab-8528-3ebcfa9c2c12-host\") pod \"node-ca-jbjh4\" (UID: \"80966b44-0240-48ab-8528-3ebcfa9c2c12\") " pod="openshift-image-registry/node-ca-jbjh4" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318726 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-hostroot\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318750 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-etc-kubernetes\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.319635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.318781 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/717175a0-02eb-40d7-b696-828a3cb7fcaa-hostroot\") pod \"multus-48mxn\" (UID: \"717175a0-02eb-40d7-b696-828a3cb7fcaa\") " pod="openshift-multus/multus-48mxn" Nov 01 14:17:42.320447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.319880 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:42.412696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.412665 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:42.423883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.423866 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 14:17:42.667935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.667860 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:42.854425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:42.854382 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:44.457430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:44.457394 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:44.473874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:44.473847 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:45.147285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:45.147250 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerStarted","Data":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} Nov 01 14:17:45.147412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:45.147347 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerStarted","Data":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} Nov 01 14:17:45.147412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:45.147388 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerStarted","Data":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} Nov 01 14:17:45.147473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:45.147423 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerStarted","Data":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} Nov 01 14:17:45.175030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:45.175000 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fx4jc" Nov 01 14:17:45.873698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:45.873666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:46.039188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:46.039154 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:46.444434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:46.444401 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:46.494137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:46.494105 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:46.605298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:46.605263 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:47.758883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:47.758843 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:17:49.815877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:49.815837 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:17:50.125615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:50.125543 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:50.125815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:50.125798 5473 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 14:17:51.455724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:51.455698 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9vsn2" Nov 01 14:17:52.147957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:52.147922 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:52.298045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:52.298014 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:52.475907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:52.475829 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:57.141479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:57.141452 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:17:57.341342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:57.341312 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:57.514028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:57.513948 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:58.833164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:58.833125 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:59.039785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:59.039754 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:59.193148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:59.193054 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:59.532878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:59.532849 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:17:59.962719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:17:59.962645 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-82phh" Nov 01 14:18:00.901803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.901770 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:18:00.902161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.902138 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="alertmanager" containerID="cri-o://2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" gracePeriod=120 Nov 01 14:18:00.902317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.902276 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy" containerID="cri-o://60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" gracePeriod=120 Nov 01 14:18:00.902389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.902313 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="config-reloader" containerID="cri-o://f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" gracePeriod=120 Nov 01 14:18:00.902389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.902335 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="prom-label-proxy" containerID="cri-o://0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" gracePeriod=120 Nov 01 14:18:00.902503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.902282 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-web" containerID="cri-o://17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" gracePeriod=120 Nov 01 14:18:00.902581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:00.902559 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-metric" containerID="cri-o://9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" gracePeriod=120 Nov 01 14:18:01.111523 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.111482 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:18:01.166247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.166182 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.203617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203585 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" exitCode=0 Nov 01 14:18:01.203617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203615 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" exitCode=0 Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203625 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" exitCode=0 Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203634 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" exitCode=0 Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203642 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" exitCode=0 Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203650 5473 generic.go:356] "Generic (PLEG): container finished" podID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" exitCode=0 Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203675 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203701 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} Nov 01 14:18:01.203780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203768 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} Nov 01 14:18:01.204018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203795 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} Nov 01 14:18:01.204018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203822 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} Nov 01 14:18:01.204018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203837 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.204018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203847 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} Nov 01 14:18:01.204018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203871 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} Nov 01 14:18:01.204018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.203895 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8a72f134-3d8c-49b8-a3f8-b874e643ca0e","Type":"ContainerDied","Data":"501807be039b39b374998332a7ff8e3a3b98d7d3dbecbb902a93fd35a911df23"} Nov 01 14:18:01.211584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.211562 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.218733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.218713 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.225634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.225616 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.231748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.231732 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.238225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.238197 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.244789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.244768 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.250966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.250937 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.251267 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.251247 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.251346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.251283 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} err="failed to get container status \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" Nov 01 14:18:01.251346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.251327 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.251571 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.251554 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.251619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.251577 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} err="failed to get container status \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" Nov 01 14:18:01.251619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.251592 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.251794 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.251775 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.251859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.251803 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} err="failed to get container status \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" Nov 01 14:18:01.251859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.251824 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.252037 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.252022 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.252090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252041 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} err="failed to get container status \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" Nov 01 14:18:01.252090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252065 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.252328 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.252311 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.252398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252336 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} err="failed to get container status \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" Nov 01 14:18:01.252398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252354 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.252601 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.252586 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.252634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252606 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} err="failed to get container status \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" Nov 01 14:18:01.252634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252619 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.252832 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:01.252816 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.252893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252839 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} err="failed to get container status \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" Nov 01 14:18:01.252893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.252857 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.253092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253074 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} err="failed to get container status \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" Nov 01 14:18:01.253136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253093 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.253312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253294 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} err="failed to get container status \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" Nov 01 14:18:01.253374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253313 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.253528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253511 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} err="failed to get container status \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" Nov 01 14:18:01.253574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253529 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.253723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253707 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} err="failed to get container status \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" Nov 01 14:18:01.253769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253723 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.253933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253916 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} err="failed to get container status \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" Nov 01 14:18:01.253992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.253934 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.254166 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254141 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} err="failed to get container status \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" Nov 01 14:18:01.254198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254166 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.254368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254352 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} err="failed to get container status \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" Nov 01 14:18:01.254417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254367 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.254571 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254550 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} err="failed to get container status \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" Nov 01 14:18:01.254571 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254569 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.254764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254749 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} err="failed to get container status \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" Nov 01 14:18:01.254764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254764 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.254964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254946 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} err="failed to get container status \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" Nov 01 14:18:01.255012 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.254965 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.255143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255127 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} err="failed to get container status \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" Nov 01 14:18:01.255190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255144 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.255395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255380 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} err="failed to get container status \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" Nov 01 14:18:01.255438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255395 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.255612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255591 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} err="failed to get container status \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" Nov 01 14:18:01.255612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255609 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.255826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255811 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} err="failed to get container status \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" Nov 01 14:18:01.255826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255826 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.255999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255985 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} err="failed to get container status \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" Nov 01 14:18:01.255999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.255998 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.256241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256206 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} err="failed to get container status \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" Nov 01 14:18:01.256317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256242 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.256469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256454 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} err="failed to get container status \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" Nov 01 14:18:01.256469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256468 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.256651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256636 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} err="failed to get container status \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" Nov 01 14:18:01.256695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256651 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.256832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256813 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} err="failed to get container status \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" Nov 01 14:18:01.256877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.256832 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.257038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257023 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} err="failed to get container status \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" Nov 01 14:18:01.257038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257036 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.257340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257323 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} err="failed to get container status \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" Nov 01 14:18:01.257412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257341 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.257565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257545 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} err="failed to get container status \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" Nov 01 14:18:01.257565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257565 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.257804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257788 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} err="failed to get container status \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" Nov 01 14:18:01.257804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257803 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.257996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257981 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} err="failed to get container status \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" Nov 01 14:18:01.258046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.257996 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.258181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258165 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} err="failed to get container status \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" Nov 01 14:18:01.258234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258181 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.258376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258360 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} err="failed to get container status \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" Nov 01 14:18:01.258376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258376 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.258568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258549 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} err="failed to get container status \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" Nov 01 14:18:01.258637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258570 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.258770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258749 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} err="failed to get container status \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" Nov 01 14:18:01.258813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258769 5473 scope.go:122] "RemoveContainer" containerID="0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f" Nov 01 14:18:01.258985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258966 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f"} err="failed to get container status \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": rpc error: code = NotFound desc = could not find container \"0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f\": container with ID starting with 0f5522ecb94a097f5e1073c927800b500845b03aa9f3dc52c2204b64427a219f not found: ID does not exist" Nov 01 14:18:01.259029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.258985 5473 scope.go:122] "RemoveContainer" containerID="9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1" Nov 01 14:18:01.259239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259208 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1"} err="failed to get container status \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": rpc error: code = NotFound desc = could not find container \"9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1\": container with ID starting with 9d6f29b180dda772369afaf795427ea247f48f1bb71f9c1ac618d14b01212ac1 not found: ID does not exist" Nov 01 14:18:01.259287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259240 5473 scope.go:122] "RemoveContainer" containerID="60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d" Nov 01 14:18:01.259447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259430 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d"} err="failed to get container status \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": rpc error: code = NotFound desc = could not find container \"60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d\": container with ID starting with 60359a35554e0f6cf87067a6bc256d459792e3505a759dbf4334d7212afd9b7d not found: ID does not exist" Nov 01 14:18:01.259492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259448 5473 scope.go:122] "RemoveContainer" containerID="17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2" Nov 01 14:18:01.259662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259646 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2"} err="failed to get container status \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": rpc error: code = NotFound desc = could not find container \"17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2\": container with ID starting with 17d50e9b228a99412c7db79ce69d40c6368343488066954ea27950b588fe9ff2 not found: ID does not exist" Nov 01 14:18:01.259710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259664 5473 scope.go:122] "RemoveContainer" containerID="f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52" Nov 01 14:18:01.259863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259846 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52"} err="failed to get container status \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": rpc error: code = NotFound desc = could not find container \"f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52\": container with ID starting with f574d74d5388f3e96ef3d63b58417840ed23feb93de082a574b0b214ac934f52 not found: ID does not exist" Nov 01 14:18:01.259897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.259863 5473 scope.go:122] "RemoveContainer" containerID="2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1" Nov 01 14:18:01.260085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.260068 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1"} err="failed to get container status \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": rpc error: code = NotFound desc = could not find container \"2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1\": container with ID starting with 2a83085ba3a35099f8aaac04dd88c5b7035b37a9e21ce6ca989c78a2f7382ec1 not found: ID does not exist" Nov 01 14:18:01.260127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.260085 5473 scope.go:122] "RemoveContainer" containerID="46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5" Nov 01 14:18:01.260309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.260294 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5"} err="failed to get container status \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": rpc error: code = NotFound desc = could not find container \"46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5\": container with ID starting with 46801eee19eddaf31f2e6f381e016246829bf05f6bece31cd800a4783770a9c5 not found: ID does not exist" Nov 01 14:18:01.271838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.271821 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.271908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.271858 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.271908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.271900 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.271972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.271933 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.271972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.271962 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272025 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272078 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272152 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:18:01.272280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272232 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272353 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272393 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272432 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272475 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.272763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272493 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:01.272763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272506 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") pod \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\" (UID: \"8a72f134-3d8c-49b8-a3f8-b874e643ca0e\") " Nov 01 14:18:01.273104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272782 5473 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-main-db\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.273104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272803 5473 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.273104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.272856 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:01.275153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.275101 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:18:01.275153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.275110 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.275331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.275228 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:18:01.275331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.275237 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.275426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.275390 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.275554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.275532 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.276174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.276148 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "kube-api-access-pd2z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:18:01.276497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.276475 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.280682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.280659 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.286418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.286393 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:18:01.286635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.286614 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config" pod "8a72f134-3d8c-49b8-a3f8-b874e643ca0e" (UID: "8a72f134-3d8c-49b8-a3f8-b874e643ca0e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:01.373713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373675 5473 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-web-config\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373709 5473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373726 5473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373746 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd2z4\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-kube-api-access-pd2z4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373762 5473 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-cluster-tls-config\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373776 5473 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373790 5473 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-metrics-client-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373806 5473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373822 5473 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-config-out\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373837 5473 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-secret-alertmanager-main-tls\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.373917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.373852 5473 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8a72f134-3d8c-49b8-a3f8-b874e643ca0e-tls-assets\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:01.616054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.616011 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:18:01.621867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.621830 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:18:01.645985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.645963 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:18:01.646276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646262 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="config-reloader" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646278 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="config-reloader" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646287 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="alertmanager" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646293 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="alertmanager" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646299 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646304 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646310 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-web" Nov 01 14:18:01.646324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646315 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-web" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646330 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-metric" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646335 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-metric" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646343 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="init-config-reloader" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646347 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="init-config-reloader" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646355 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="prom-label-proxy" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646359 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="prom-label-proxy" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646397 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="config-reloader" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646404 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="alertmanager" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646409 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-metric" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646415 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646427 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="kube-rbac-proxy-web" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646432 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="init-config-reloader" Nov 01 14:18:01.646558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.646438 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" containerName="prom-label-proxy" Nov 01 14:18:01.652009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.651990 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676446 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676486 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676503 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/496d04ed-d532-4793-a407-e0cea3c23e2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676518 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676534 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676555 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztljh\" (UniqueName: \"kubernetes.io/projected/496d04ed-d532-4793-a407-e0cea3c23e2a-kube-api-access-ztljh\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676702 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/496d04ed-d532-4793-a407-e0cea3c23e2a-config-out\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676816 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676885 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676874 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/496d04ed-d532-4793-a407-e0cea3c23e2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676914 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496d04ed-d532-4793-a407-e0cea3c23e2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676946 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.676969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676962 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/496d04ed-d532-4793-a407-e0cea3c23e2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.677063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.676981 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-web-config\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701405 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701410 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701456 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701475 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701486 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701506 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 14:18:01.701570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701512 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vd72s\"" Nov 01 14:18:01.701913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701578 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 14:18:01.701913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.701753 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 14:18:01.702082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.702064 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 14:18:01.747757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.747728 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:18:01.777470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777404 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777452 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777470 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/496d04ed-d532-4793-a407-e0cea3c23e2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777512 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777546 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777738 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztljh\" (UniqueName: \"kubernetes.io/projected/496d04ed-d532-4793-a407-e0cea3c23e2a-kube-api-access-ztljh\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777797 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/496d04ed-d532-4793-a407-e0cea3c23e2a-config-out\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777802 5473 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 14:18:01.777844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777832 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777862 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/496d04ed-d532-4793-a407-e0cea3c23e2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777906 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496d04ed-d532-4793-a407-e0cea3c23e2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777936 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777957 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/496d04ed-d532-4793-a407-e0cea3c23e2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.777995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.777982 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-web-config\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.778934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.778905 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/496d04ed-d532-4793-a407-e0cea3c23e2a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.779267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.779237 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/496d04ed-d532-4793-a407-e0cea3c23e2a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.779691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.779665 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/496d04ed-d532-4793-a407-e0cea3c23e2a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.780721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.780695 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/496d04ed-d532-4793-a407-e0cea3c23e2a-config-out\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.780812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.780752 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.780852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.780840 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.781411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.781388 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-web-config\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.781659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.781621 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.781659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.781652 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.781793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.781728 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.781919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.781899 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/496d04ed-d532-4793-a407-e0cea3c23e2a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.782650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.782627 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/496d04ed-d532-4793-a407-e0cea3c23e2a-config-volume\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.830100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.830064 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztljh\" (UniqueName: \"kubernetes.io/projected/496d04ed-d532-4793-a407-e0cea3c23e2a-kube-api-access-ztljh\") pod \"alertmanager-main-0\" (UID: \"496d04ed-d532-4793-a407-e0cea3c23e2a\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:01.978502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:01.978463 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 14:18:02.388578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:02.388541 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 14:18:02.391729 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:18:02.391701 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496d04ed_d532_4793_a407_e0cea3c23e2a.slice/crio-7540cf6eaf849e11009dcdbf46fb3630c581be382d45cd5c8790f37950dda791 WatchSource:0}: Error finding container 7540cf6eaf849e11009dcdbf46fb3630c581be382d45cd5c8790f37950dda791: Status 404 returned error can't find the container with id 7540cf6eaf849e11009dcdbf46fb3630c581be382d45cd5c8790f37950dda791 Nov 01 14:18:03.079619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:03.079587 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a72f134-3d8c-49b8-a3f8-b874e643ca0e" path="/var/lib/kubelet/pods/8a72f134-3d8c-49b8-a3f8-b874e643ca0e/volumes" Nov 01 14:18:03.212770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:03.212739 5473 generic.go:356] "Generic (PLEG): container finished" podID="496d04ed-d532-4793-a407-e0cea3c23e2a" containerID="839742729b21ef873af8e2affd7ecd2c4e69e2e7e00f918a2a8df24cddbc9fee" exitCode=0 Nov 01 14:18:03.212963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:03.212809 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerDied","Data":"839742729b21ef873af8e2affd7ecd2c4e69e2e7e00f918a2a8df24cddbc9fee"} Nov 01 14:18:03.212963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:03.212858 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"7540cf6eaf849e11009dcdbf46fb3630c581be382d45cd5c8790f37950dda791"} Nov 01 14:18:04.220020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.219984 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"f4277991236c4d22fafbcc051eee810a94368804596e8c04dc99c6da01677459"} Nov 01 14:18:04.220511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.220081 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"0736c98dfd709b8748aa4ec5d2bde98bcb31db816e1d5fca4f398d3f7b80b8b4"} Nov 01 14:18:04.220511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.220111 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"5503fff18bc68069b3fad4696476e4c565e0126afe56f43dddb1ffb5294aaf6b"} Nov 01 14:18:04.220511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.220134 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"68e4a421b614b41b223d757cdb44dea0ecafa21ce048420ce6e2e9bd3cc51313"} Nov 01 14:18:04.220511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.220156 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"9fa288824090cc1a8ff5ba3c01fd2164c31cc517a5f8e45f7b4d68d6443d620a"} Nov 01 14:18:04.220511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.220179 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"496d04ed-d532-4793-a407-e0cea3c23e2a","Type":"ContainerStarted","Data":"af8902a04ba38f1bd03405f7a21e5953177ceef6529633071e048c8d7ae26e29"} Nov 01 14:18:04.362139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.362085 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.349848466 podStartE2EDuration="3.362069471s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.221005ms" isStatefulPod=false podCreationTimestamp="2026-11-01 14:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:18:04.360927552 +0000 UTC m=+23.419079403" watchObservedRunningTime="2026-11-01 14:18:04.362069471 +0000 UTC m=+23.420221321" Nov 01 14:18:04.682487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.682450 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c9bbb4584-9lxxr"] Nov 01 14:18:04.686105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.686084 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.734889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.734851 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 14:18:04.735095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.734958 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 14:18:04.735095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.734969 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 14:18:04.735095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.734981 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 14:18:04.735095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.735069 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w9v8n\"" Nov 01 14:18:04.735281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.735161 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 14:18:04.805708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805682 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-metrics-client-ca\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.805850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805715 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-federate-client-tls\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.805850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805742 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-secret-telemeter-client\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.805850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805834 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.805953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805886 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.805953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805906 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-serving-certs-ca-bundle\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.805953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805935 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdc7\" (UniqueName: \"kubernetes.io/projected/aff6a7b6-cf63-43c0-ba97-06900306308d-kube-api-access-5sdc7\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.806040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.805997 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-telemeter-client-tls\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.826708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.826671 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 14:18:04.827528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.827497 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c9bbb4584-9lxxr"] Nov 01 14:18:04.906980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.906950 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.906980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.906985 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-serving-certs-ca-bundle\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907015 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdc7\" (UniqueName: \"kubernetes.io/projected/aff6a7b6-cf63-43c0-ba97-06900306308d-kube-api-access-5sdc7\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907042 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-telemeter-client-tls\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907076 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-metrics-client-ca\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907093 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-federate-client-tls\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907116 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-secret-telemeter-client\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907168 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907874 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-serving-certs-ca-bundle\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.907904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907897 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-metrics-client-ca\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.908082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.907992 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aff6a7b6-cf63-43c0-ba97-06900306308d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.909718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.909697 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-federate-client-tls\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.909815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.909734 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-telemeter-client-tls\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.910141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.910118 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.910173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.910138 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/aff6a7b6-cf63-43c0-ba97-06900306308d-secret-telemeter-client\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:04.959189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:04.959123 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdc7\" (UniqueName: \"kubernetes.io/projected/aff6a7b6-cf63-43c0-ba97-06900306308d-kube-api-access-5sdc7\") pod \"telemeter-client-c9bbb4584-9lxxr\" (UID: \"aff6a7b6-cf63-43c0-ba97-06900306308d\") " pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:05.003188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:05.003162 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" Nov 01 14:18:05.414014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:05.413975 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c9bbb4584-9lxxr"] Nov 01 14:18:05.417772 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:18:05.417733 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff6a7b6_cf63_43c0_ba97_06900306308d.slice/crio-e246e92ce7eec4780e08ef7a4bc738630b6ce9244411cfcb54532e6a7781f91f WatchSource:0}: Error finding container e246e92ce7eec4780e08ef7a4bc738630b6ce9244411cfcb54532e6a7781f91f: Status 404 returned error can't find the container with id e246e92ce7eec4780e08ef7a4bc738630b6ce9244411cfcb54532e6a7781f91f Nov 01 14:18:06.227234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.227190 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" event={"ID":"aff6a7b6-cf63-43c0-ba97-06900306308d","Type":"ContainerStarted","Data":"e246e92ce7eec4780e08ef7a4bc738630b6ce9244411cfcb54532e6a7781f91f"} Nov 01 14:18:06.591132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.591097 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:18:06.593189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.593129 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="prometheus" containerID="cri-o://13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" gracePeriod=600 Nov 01 14:18:06.593472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.593204 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="thanos-sidecar" containerID="cri-o://69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" gracePeriod=600 Nov 01 14:18:06.593689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.593200 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" gracePeriod=600 Nov 01 14:18:06.594154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.593179 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy" containerID="cri-o://b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" gracePeriod=600 Nov 01 14:18:06.594154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.593250 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-web" containerID="cri-o://efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" gracePeriod=600 Nov 01 14:18:06.594154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.593266 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="config-reloader" containerID="cri-o://1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" gracePeriod=600 Nov 01 14:18:06.864210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.864182 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:06.924654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924624 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924656 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924684 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924708 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924728 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924754 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924777 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924798 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924824 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.924875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924855 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924887 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924920 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.924959 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925042 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925080 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925111 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925151 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925179 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925246 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925275 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") pod \"d8121105-7c56-42bf-89a2-858453626862\" (UID: \"d8121105-7c56-42bf-89a2-858453626862\") " Nov 01 14:18:06.925792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925600 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.925792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.925699 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:18:06.926605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.926339 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.927256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.926930 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.927256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.926968 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.927416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.927298 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.927751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.927727 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.927826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.927740 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:18:06.927990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.927963 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:18:06.928118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.928081 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.928187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.928162 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.928431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.928393 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.928516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.928437 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.928516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.928482 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.929695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.929671 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.929798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.929738 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "kube-api-access-p9pdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:18:06.929942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.929918 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.930192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.930164 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:06.930339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.930189 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:18:06.938528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:06.938507 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config" pod "d8121105-7c56-42bf-89a2-858453626862" (UID: "d8121105-7c56-42bf-89a2-858453626862"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:18:07.026289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026253 5473 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-db\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026285 5473 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026300 5473 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-kube-rbac-proxy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026313 5473 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-config\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026323 5473 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026336 5473 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026351 5473 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026366 5473 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-tls-assets\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026380 5473 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026395 5473 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026411 5473 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026427 5473 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-web-config\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026441 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pdt\" (UniqueName: \"kubernetes.io/projected/d8121105-7c56-42bf-89a2-858453626862-kube-api-access-p9pdt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026455 5473 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-metrics-client-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026471 5473 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026485 5473 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026500 5473 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d8121105-7c56-42bf-89a2-858453626862-config-out\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026514 5473 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-grpc-tls\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026526 5473 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8121105-7c56-42bf-89a2-858453626862-configmap-metrics-client-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.026944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.026535 5473 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d8121105-7c56-42bf-89a2-858453626862-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:18:07.145073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.144995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:07.236029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.235993 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" exitCode=0 Nov 01 14:18:07.236029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236021 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" exitCode=0 Nov 01 14:18:07.236029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236032 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" exitCode=0 Nov 01 14:18:07.236029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236042 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" exitCode=0 Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236050 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" exitCode=0 Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236066 5473 generic.go:356] "Generic (PLEG): container finished" podID="d8121105-7c56-42bf-89a2-858453626862" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" exitCode=0 Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236084 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236128 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236163 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236197 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236249 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.236395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236250 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} Nov 01 14:18:07.236730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236417 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} Nov 01 14:18:07.236730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236464 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} Nov 01 14:18:07.236730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.236506 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d8121105-7c56-42bf-89a2-858453626862","Type":"ContainerDied","Data":"9d93d0e5e13587c7a68d1ac965dbaadfc50fabd4144a47d94aeee31145be575f"} Nov 01 14:18:07.244675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.244655 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.251387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.251358 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.259432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.259410 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.267151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.267132 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.273660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.273638 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.281585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.281561 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.288793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.288775 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.289068 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.289047 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.289141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289081 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} err="failed to get container status \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" Nov 01 14:18:07.289141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289106 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.289358 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.289342 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.289426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289366 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} err="failed to get container status \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" Nov 01 14:18:07.289426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289386 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.289693 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.289674 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.289735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289701 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} err="failed to get container status \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" Nov 01 14:18:07.289735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289718 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.289967 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.289950 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.290083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289975 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} err="failed to get container status \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" Nov 01 14:18:07.290083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.289994 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.290240 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.290209 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.290280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290241 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} err="failed to get container status \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" Nov 01 14:18:07.290280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290255 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.290502 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.290484 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.290569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290509 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} err="failed to get container status \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" Nov 01 14:18:07.290569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290528 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.290768 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:07.290752 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.290828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290776 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} err="failed to get container status \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" Nov 01 14:18:07.290828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290796 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.291019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.290998 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} err="failed to get container status \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" Nov 01 14:18:07.291085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291020 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.291278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291258 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} err="failed to get container status \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" Nov 01 14:18:07.291326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291279 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.291499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291473 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} err="failed to get container status \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" Nov 01 14:18:07.291576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291500 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.291756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291738 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} err="failed to get container status \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" Nov 01 14:18:07.291756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291755 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.291999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291978 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} err="failed to get container status \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" Nov 01 14:18:07.291999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.291997 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.292240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292206 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} err="failed to get container status \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" Nov 01 14:18:07.292240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292239 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.292467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292448 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} err="failed to get container status \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" Nov 01 14:18:07.292467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292466 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.292671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292654 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} err="failed to get container status \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" Nov 01 14:18:07.292671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292670 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.292892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292865 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} err="failed to get container status \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" Nov 01 14:18:07.292892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.292881 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.293099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293066 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} err="failed to get container status \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" Nov 01 14:18:07.293099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293090 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.293324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293302 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} err="failed to get container status \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" Nov 01 14:18:07.293324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293319 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.293561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293543 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} err="failed to get container status \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" Nov 01 14:18:07.293611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293561 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.293803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293785 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} err="failed to get container status \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" Nov 01 14:18:07.293803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.293802 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.294146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294112 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} err="failed to get container status \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" Nov 01 14:18:07.294257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294147 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.294443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294416 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} err="failed to get container status \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" Nov 01 14:18:07.294443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294435 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.294694 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294669 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} err="failed to get container status \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" Nov 01 14:18:07.294786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294692 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.294966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294943 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} err="failed to get container status \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" Nov 01 14:18:07.295031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.294967 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.295256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.295232 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} err="failed to get container status \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" Nov 01 14:18:07.295316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.295257 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.295543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.295519 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} err="failed to get container status \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" Nov 01 14:18:07.295543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.295541 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.295768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.295751 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} err="failed to get container status \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" Nov 01 14:18:07.295768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.295768 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.296034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296017 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} err="failed to get container status \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" Nov 01 14:18:07.296034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296033 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.296333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296317 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} err="failed to get container status \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" Nov 01 14:18:07.296376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296334 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.296577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296554 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} err="failed to get container status \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" Nov 01 14:18:07.296628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296577 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.296798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296780 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} err="failed to get container status \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" Nov 01 14:18:07.296798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296797 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.297025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.296998 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} err="failed to get container status \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" Nov 01 14:18:07.297089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297027 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.297264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297246 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} err="failed to get container status \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" Nov 01 14:18:07.297310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297264 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.297469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297453 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} err="failed to get container status \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" Nov 01 14:18:07.297527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297468 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.297666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297650 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} err="failed to get container status \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" Nov 01 14:18:07.297709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297666 5473 scope.go:122] "RemoveContainer" containerID="4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d" Nov 01 14:18:07.297886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297867 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d"} err="failed to get container status \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": rpc error: code = NotFound desc = could not find container \"4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d\": container with ID starting with 4f0cab2d5864f0b97bd7007f305a2d4bbf8288b541db42dd35d92f1c001d2d1d not found: ID does not exist" Nov 01 14:18:07.297951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.297888 5473 scope.go:122] "RemoveContainer" containerID="b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3" Nov 01 14:18:07.298115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.298097 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3"} err="failed to get container status \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": rpc error: code = NotFound desc = could not find container \"b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3\": container with ID starting with b398c9ed4ac657a9756c8072e03b877c2016f90ad1c1d15e07a9cc3cad8e5fb3 not found: ID does not exist" Nov 01 14:18:07.298183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.298115 5473 scope.go:122] "RemoveContainer" containerID="efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294" Nov 01 14:18:07.298390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.298366 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294"} err="failed to get container status \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": rpc error: code = NotFound desc = could not find container \"efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294\": container with ID starting with efa2d97bdb8a7fda00465983db611282d6b459dff13984a43a3461bebc811294 not found: ID does not exist" Nov 01 14:18:07.298390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.298389 5473 scope.go:122] "RemoveContainer" containerID="69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4" Nov 01 14:18:07.298818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.298784 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4"} err="failed to get container status \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": rpc error: code = NotFound desc = could not find container \"69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4\": container with ID starting with 69cea54ea74f1bb791fcfb2f7994013d1922deaac39337a1bbb87b976026b8d4 not found: ID does not exist" Nov 01 14:18:07.298818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.298806 5473 scope.go:122] "RemoveContainer" containerID="1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51" Nov 01 14:18:07.299085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.299066 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51"} err="failed to get container status \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": rpc error: code = NotFound desc = could not find container \"1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51\": container with ID starting with 1425a0b0a509dd63ee78d72e1ce6122455ce7fb70e83c299b7306a64cab70b51 not found: ID does not exist" Nov 01 14:18:07.299165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.299086 5473 scope.go:122] "RemoveContainer" containerID="13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43" Nov 01 14:18:07.299393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.299371 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43"} err="failed to get container status \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": rpc error: code = NotFound desc = could not find container \"13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43\": container with ID starting with 13f51c00e9d7ff8f0aab7e5b94e45af83c5ee51b9dff976ed1465c3a447d2c43 not found: ID does not exist" Nov 01 14:18:07.299454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.299396 5473 scope.go:122] "RemoveContainer" containerID="2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d" Nov 01 14:18:07.299654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.299629 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d"} err="failed to get container status \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": rpc error: code = NotFound desc = could not find container \"2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d\": container with ID starting with 2bfc3df4ff885b6ff0a5eb7914549daa76865cfde5de4935156b2498df66222d not found: ID does not exist" Nov 01 14:18:07.321579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.321557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:07.390974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.390938 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:18:07.436879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.436799 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:18:07.437409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437384 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:18:07.437764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437745 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-web" Nov 01 14:18:07.437836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437765 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-web" Nov 01 14:18:07.437836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437790 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy" Nov 01 14:18:07.437836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437799 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy" Nov 01 14:18:07.437836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437813 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="prometheus" Nov 01 14:18:07.437836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437821 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="prometheus" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437842 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="init-config-reloader" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437850 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="init-config-reloader" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437863 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="config-reloader" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437870 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="config-reloader" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437881 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="thanos-sidecar" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437889 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="thanos-sidecar" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437899 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-thanos" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437906 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-thanos" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437962 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="config-reloader" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437973 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="thanos-sidecar" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437982 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-thanos" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.437993 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy-web" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.438003 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="kube-rbac-proxy" Nov 01 14:18:07.438007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.438012 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8121105-7c56-42bf-89a2-858453626862" containerName="prometheus" Nov 01 14:18:07.443415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.443389 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.494288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.494085 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 14:18:07.494288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.494164 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 14:18:07.494288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.494147 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 14:18:07.494288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.494256 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 14:18:07.495275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.494336 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 14:18:07.495275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495271 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 14:18:07.495477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495447 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 14:18:07.495594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495475 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 14:18:07.495594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495520 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 14:18:07.495594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495521 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gvptn\"" Nov 01 14:18:07.495594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495574 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 14:18:07.495801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495782 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 14:18:07.495892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495802 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-29vjcn8bfe0n4\"" Nov 01 14:18:07.496007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.495987 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 14:18:07.531133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531099 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a82d731-8a69-4a50-8246-8e3ac6363720-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531145 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a82d731-8a69-4a50-8246-8e3ac6363720-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531173 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531285 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531310 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531373 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531392 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531409 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtpz2\" (UniqueName: \"kubernetes.io/projected/1a82d731-8a69-4a50-8246-8e3ac6363720-kube-api-access-xtpz2\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531436 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531469 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-config\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531551 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531653 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531690 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531762 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531786 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531802 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531837 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531858 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.531890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531875 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.532038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.531890 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.541510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.541488 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 14:18:07.541645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.541528 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 14:18:07.558310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.558284 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:18:07.635622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635589 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a82d731-8a69-4a50-8246-8e3ac6363720-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635629 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a82d731-8a69-4a50-8246-8e3ac6363720-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635654 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635676 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635703 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635742 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635772 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635799 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtpz2\" (UniqueName: \"kubernetes.io/projected/1a82d731-8a69-4a50-8246-8e3ac6363720-kube-api-access-xtpz2\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635829 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635852 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-config\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635881 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635926 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635959 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.635990 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.636018 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.636041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.636086 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.636117 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.636143 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.636913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.636167 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.638683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.637576 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.638683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.638248 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.640411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.640388 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.640584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.640565 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.642337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.641268 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.642337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.642003 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.647525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.646774 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-config\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.647525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.647267 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.649416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.649146 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a82d731-8a69-4a50-8246-8e3ac6363720-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.651151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.651128 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.651628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.651606 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a82d731-8a69-4a50-8246-8e3ac6363720-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.654440 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.654417 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a82d731-8a69-4a50-8246-8e3ac6363720-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.654816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.654794 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.655295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.655268 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.655736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.655704 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.655816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.655745 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.656175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.656154 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.656300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.656280 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.657274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.657252 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a82d731-8a69-4a50-8246-8e3ac6363720-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.690256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.690173 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtpz2\" (UniqueName: \"kubernetes.io/projected/1a82d731-8a69-4a50-8246-8e3ac6363720-kube-api-access-xtpz2\") pod \"prometheus-k8s-0\" (UID: \"1a82d731-8a69-4a50-8246-8e3ac6363720\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.733196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.733164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:07.777584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.777554 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:07.892486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:07.892462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:08.242077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:08.241977 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" event={"ID":"aff6a7b6-cf63-43c0-ba97-06900306308d","Type":"ContainerStarted","Data":"89474d3520765ee5f2541b221c9107db234ce9b785b61cb0fa9eaf299a7252d5"} Nov 01 14:18:08.242077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:08.242050 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" event={"ID":"aff6a7b6-cf63-43c0-ba97-06900306308d","Type":"ContainerStarted","Data":"0464995418af682ad6d2a8817e707e3bcf3d23d5dcd3c9243640c8c8e67072f6"} Nov 01 14:18:08.242077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:08.242078 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" event={"ID":"aff6a7b6-cf63-43c0-ba97-06900306308d","Type":"ContainerStarted","Data":"028f746861510a3a616b68a9a89ee9fb25796ff808eebb6f9f429a697aaeee99"} Nov 01 14:18:08.247833 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:08.247810 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 14:18:08.249348 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:18:08.249324 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a82d731_8a69_4a50_8246_8e3ac6363720.slice/crio-8677b9908ea98fb83444f4da1d4f3d55ecacf5fc7b06cf8ba5a9f101cf64b072 WatchSource:0}: Error finding container 8677b9908ea98fb83444f4da1d4f3d55ecacf5fc7b06cf8ba5a9f101cf64b072: Status 404 returned error can't find the container with id 8677b9908ea98fb83444f4da1d4f3d55ecacf5fc7b06cf8ba5a9f101cf64b072 Nov 01 14:18:08.351861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:08.351815 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c9bbb4584-9lxxr" podStartSLOduration=1.910595147 podStartE2EDuration="4.351800457s" totalImagesPullingTime="2.44120531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:18:08.350420642 +0000 UTC m=+27.408572493" watchObservedRunningTime="2026-11-01 14:18:08.351800457 +0000 UTC m=+27.409952306" Nov 01 14:18:09.078816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:09.078783 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8121105-7c56-42bf-89a2-858453626862" path="/var/lib/kubelet/pods/d8121105-7c56-42bf-89a2-858453626862/volumes" Nov 01 14:18:09.247439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:09.247398 5473 generic.go:356] "Generic (PLEG): container finished" podID="1a82d731-8a69-4a50-8246-8e3ac6363720" containerID="3cefdff72f5ed19fd31967de076e4129e471ca468920ed1d3b61f4aa1f3c887b" exitCode=0 Nov 01 14:18:09.247600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:09.247488 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerDied","Data":"3cefdff72f5ed19fd31967de076e4129e471ca468920ed1d3b61f4aa1f3c887b"} Nov 01 14:18:09.247600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:09.247552 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"8677b9908ea98fb83444f4da1d4f3d55ecacf5fc7b06cf8ba5a9f101cf64b072"} Nov 01 14:18:09.818939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:09.818910 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-t7sh8" Nov 01 14:18:10.254900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.254871 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:18:10.255356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255249 5473 generic.go:356] "Generic (PLEG): container finished" podID="1a82d731-8a69-4a50-8246-8e3ac6363720" containerID="5f09e03e12a27ec65e2d8b02e8b6d361cd7257c006e632bb15d7da6f18f713b2" exitCode=2 Nov 01 14:18:10.255356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255330 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"5ec774ba089cdbe2464a5bc1be8ddcb51a9a5945e601229dd7ccd32dbe8eba68"} Nov 01 14:18:10.255451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255382 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"819a6a32c4cdda3447753ebe14efc0b2aa64f55326594683d1e6e6697058b9e6"} Nov 01 14:18:10.255451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255411 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"8d7a114d6fe1a4ffe8cd45989faec7ff51ef59c52223b7a58aec095a4649d1d2"} Nov 01 14:18:10.255451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255437 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"b16a6852513d9e888cd0e07e80b87f8325dc7985a0a16dc20a25a615e5ca6808"} Nov 01 14:18:10.255593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255461 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"2e599aac583ff7cc69f878e806446c4f3db004ac70d472a0f34adf7fb72e7224"} Nov 01 14:18:10.255593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255485 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerDied","Data":"5f09e03e12a27ec65e2d8b02e8b6d361cd7257c006e632bb15d7da6f18f713b2"} Nov 01 14:18:10.255871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:10.255785 5473 scope.go:122] "RemoveContainer" containerID="5f09e03e12a27ec65e2d8b02e8b6d361cd7257c006e632bb15d7da6f18f713b2" Nov 01 14:18:11.261948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:11.261919 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:18:11.262405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:11.262375 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a82d731-8a69-4a50-8246-8e3ac6363720","Type":"ContainerStarted","Data":"87700c837bb027ff92b94d6e0d01256610a283c933bd481050368291a017d54a"} Nov 01 14:18:11.423611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:11.423553 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.404045599 podStartE2EDuration="4.423539252s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.493653ms" isStatefulPod=false podCreationTimestamp="2026-11-01 14:18:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:18:11.422848631 +0000 UTC m=+30.481000480" watchObservedRunningTime="2026-11-01 14:18:11.423539252 +0000 UTC m=+30.481691101" Nov 01 14:18:12.777748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:12.777715 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:17.135767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:17.135704 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:17.288537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:17.288504 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:21.291383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:21.291357 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:18:21.295038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:21.295015 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-674fc8bc5b-hm9rs" Nov 01 14:18:22.141317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:22.141282 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:22.295386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:22.295344 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:31.105629 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:18:31.105591 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:18:42.974974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:42.974934 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:51.194243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:51.194191 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:51.211115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:51.211090 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:51.416997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:51.416971 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 14:18:56.493662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:56.493627 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:18:56.653604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:18:56.653572 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:01.105434 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:19:01.105395 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:19:09.126332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:09.126297 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:09.286752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:09.286718 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:12.081560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:12.081522 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:16.749569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:16.749529 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:16.905249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:16.905193 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:17.124814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:17.124778 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:17.293597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:17.293562 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:22.137435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:22.137337 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:22.300994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:22.300961 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:19:31.105603 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:19:31.105565 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:19:43.289697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:19:43.289648 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:20:01.106345 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:20:01.106304 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:20:08.883547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:20:08.883512 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:20:31.106726 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:20:31.106686 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:20:43.606503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:20:43.606470 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:01.106545 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:21:01.106462 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:21:19.438696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:19.438654 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:21.125962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.125924 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp"] Nov 01 14:21:21.129258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.129239 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.172605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.172571 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.172823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.172795 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmtz\" (UniqueName: \"kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.172914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.172893 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.223405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.223374 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 14:21:21.223577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.223420 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 14:21:21.223577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.223537 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-gkzc5\"" Nov 01 14:21:21.224402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.224380 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp"] Nov 01 14:21:21.227486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.227466 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t"] Nov 01 14:21:21.232326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.232303 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.273735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.273700 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmtz\" (UniqueName: \"kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.273735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.273736 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.273918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.273758 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.273918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.273787 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxh22\" (UniqueName: \"kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.273997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.273929 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.274033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.273989 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.274246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.274210 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.274298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.274262 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.280497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.280481 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rcpb5\"" Nov 01 14:21:21.289607 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.289584 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t"] Nov 01 14:21:21.324809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.324781 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8"] Nov 01 14:21:21.328331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.328314 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.330704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.330676 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmtz\" (UniqueName: \"kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.374895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.374856 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.374895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.374901 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxh22\" (UniqueName: \"kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.375116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.374934 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.375116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.374954 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.375116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.374991 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kscdg\" (UniqueName: \"kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.375116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.375011 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.375274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.375246 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.375308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.375287 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.378753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.378707 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-d7pxb\"" Nov 01 14:21:21.387354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.387332 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8"] Nov 01 14:21:21.402051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.402026 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:21.429399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.429371 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr"] Nov 01 14:21:21.432851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.432827 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.433114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.433093 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxh22\" (UniqueName: \"kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.444739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.444617 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:21:21.475494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475454 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.475712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475507 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.475712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475607 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.475712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475644 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmjc\" (UniqueName: \"kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.475712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475675 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kscdg\" (UniqueName: \"kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.475712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475703 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.475996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.475945 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.476046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.476005 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.481069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.481044 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-r2bw2\"" Nov 01 14:21:21.490419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.490390 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr"] Nov 01 14:21:21.528929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.528900 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kscdg\" (UniqueName: \"kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.549815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.549779 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:21:21.576505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.576477 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.576673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.576536 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.576673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.576558 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmjc\" (UniqueName: \"kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.576910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.576888 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.576976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.576947 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.578368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.578344 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:21.629184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.629113 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmjc\" (UniqueName: \"kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.650056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.650022 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:21:21.755438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.755407 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:21:21.807024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.806992 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp"] Nov 01 14:21:21.811504 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:21.811452 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a80f7f_10df_4f13_9c40_9307098830ab.slice/crio-b41e8fa0e1c36ca66a014f2246e329f0394de9043d8ec0d57e89b98e72d16bca WatchSource:0}: Error finding container b41e8fa0e1c36ca66a014f2246e329f0394de9043d8ec0d57e89b98e72d16bca: Status 404 returned error can't find the container with id b41e8fa0e1c36ca66a014f2246e329f0394de9043d8ec0d57e89b98e72d16bca Nov 01 14:21:21.907333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.907143 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t"] Nov 01 14:21:21.909725 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:21.909694 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a3003d_f42d_40a6_bd2c_bfdcc254b8b0.slice/crio-4d2cf288e2ed7d63158ab2a3606ca115167cce286bc50718ccf193eafc9b0deb WatchSource:0}: Error finding container 4d2cf288e2ed7d63158ab2a3606ca115167cce286bc50718ccf193eafc9b0deb: Status 404 returned error can't find the container with id 4d2cf288e2ed7d63158ab2a3606ca115167cce286bc50718ccf193eafc9b0deb Nov 01 14:21:21.915037 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.915014 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" event={"ID":"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0","Type":"ContainerStarted","Data":"4d2cf288e2ed7d63158ab2a3606ca115167cce286bc50718ccf193eafc9b0deb"} Nov 01 14:21:21.916095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:21.916074 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" event={"ID":"19a80f7f-10df-4f13-9c40-9307098830ab","Type":"ContainerStarted","Data":"b41e8fa0e1c36ca66a014f2246e329f0394de9043d8ec0d57e89b98e72d16bca"} Nov 01 14:21:22.009705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.009676 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8"] Nov 01 14:21:22.012381 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:22.012353 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938fed11_b55f_4b16_a1d2_a54382b86bfd.slice/crio-5f7b483f27bae7e45175fc287eb53e5dfdd1648d819ec20cef2ad1c78979eb4a WatchSource:0}: Error finding container 5f7b483f27bae7e45175fc287eb53e5dfdd1648d819ec20cef2ad1c78979eb4a: Status 404 returned error can't find the container with id 5f7b483f27bae7e45175fc287eb53e5dfdd1648d819ec20cef2ad1c78979eb4a Nov 01 14:21:22.113100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.113064 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr"] Nov 01 14:21:22.116072 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:22.116046 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70162213_073f_4956_a023_023ac5e753e3.slice/crio-99aac05f7658341fbb8224f5bbecef17669cccb9fcf1d23a6bfa3f71d227c889 WatchSource:0}: Error finding container 99aac05f7658341fbb8224f5bbecef17669cccb9fcf1d23a6bfa3f71d227c889: Status 404 returned error can't find the container with id 99aac05f7658341fbb8224f5bbecef17669cccb9fcf1d23a6bfa3f71d227c889 Nov 01 14:21:22.480432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.480398 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm"] Nov 01 14:21:22.484965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.484941 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.533434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.533404 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-v75nt\"" Nov 01 14:21:22.541355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.541324 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm"] Nov 01 14:21:22.585959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.585923 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8t8g\" (UniqueName: \"kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.586127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.585980 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.586169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.586124 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.687235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.687182 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8t8g\" (UniqueName: \"kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.687405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.687243 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.687405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.687280 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.687623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.687608 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.687660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.687640 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.740333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.740262 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8t8g\" (UniqueName: \"kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.801141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.801112 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:21:22.921942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.921903 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" event={"ID":"70162213-073f-4956-a023-023ac5e753e3","Type":"ContainerStarted","Data":"99aac05f7658341fbb8224f5bbecef17669cccb9fcf1d23a6bfa3f71d227c889"} Nov 01 14:21:22.923352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:22.923315 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" event={"ID":"938fed11-b55f-4b16-a1d2-a54382b86bfd","Type":"ContainerStarted","Data":"5f7b483f27bae7e45175fc287eb53e5dfdd1648d819ec20cef2ad1c78979eb4a"} Nov 01 14:21:23.165915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:23.165876 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm"] Nov 01 14:21:23.206540 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:23.206508 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9f2d61_35fd_4fe4_9db1_d320c0f8de8d.slice/crio-f5432f0041c51f504b12019f8bc23a73c6a62a9a2c2ff556e188717b6e8cbf4e WatchSource:0}: Error finding container f5432f0041c51f504b12019f8bc23a73c6a62a9a2c2ff556e188717b6e8cbf4e: Status 404 returned error can't find the container with id f5432f0041c51f504b12019f8bc23a73c6a62a9a2c2ff556e188717b6e8cbf4e Nov 01 14:21:23.931054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:23.931016 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" event={"ID":"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d","Type":"ContainerStarted","Data":"f5432f0041c51f504b12019f8bc23a73c6a62a9a2c2ff556e188717b6e8cbf4e"} Nov 01 14:21:24.058989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.058954 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79"] Nov 01 14:21:24.062887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.062862 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.160495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.159167 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-85kvn\"" Nov 01 14:21:24.160495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.160378 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79"] Nov 01 14:21:24.204919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.201437 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.204919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.201550 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.204919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.201689 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssddx\" (UniqueName: \"kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.303986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.302707 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.303986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.302774 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssddx\" (UniqueName: \"kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.303986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.302878 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.303986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.303375 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.303986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.303644 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.364272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.364142 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssddx\" (UniqueName: \"kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.385422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.385150 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:21:24.779802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.779763 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79"] Nov 01 14:21:24.819825 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:24.819785 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f87c7b_f7af_4eac_b236_ed27e59a6922.slice/crio-b91272d088e6521bbb422ca674e30f8070dd5083e1482957d89938b9b3b5b3ac WatchSource:0}: Error finding container b91272d088e6521bbb422ca674e30f8070dd5083e1482957d89938b9b3b5b3ac: Status 404 returned error can't find the container with id b91272d088e6521bbb422ca674e30f8070dd5083e1482957d89938b9b3b5b3ac Nov 01 14:21:24.937890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:24.937847 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" event={"ID":"35f87c7b-f7af-4eac-b236-ed27e59a6922","Type":"ContainerStarted","Data":"b91272d088e6521bbb422ca674e30f8070dd5083e1482957d89938b9b3b5b3ac"} Nov 01 14:21:27.110305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:27.110270 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:27.277302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:27.277262 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:27.610082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:27.609987 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:27.889824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:27.889703 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:28.056288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.056183 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:28.656238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.656186 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn"] Nov 01 14:21:28.660381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.660355 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.683022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.682991 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t"] Nov 01 14:21:28.689238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.687461 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.730290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.727884 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7"] Nov 01 14:21:28.732525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.732060 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.733924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.733873 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn"] Nov 01 14:21:28.746966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.746935 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.747087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.746980 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.747087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.747036 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.747087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.747072 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.747276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.747160 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzch\" (UniqueName: \"kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.747276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.747187 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tqk\" (UniqueName: \"kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.757032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.757001 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t"] Nov 01 14:21:28.772730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.772701 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7"] Nov 01 14:21:28.848164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848116 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzch\" (UniqueName: \"kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.848366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848169 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tqk\" (UniqueName: \"kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.848366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848230 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.848366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848263 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.848366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848319 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.848366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848359 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.848634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848391 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.848634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848442 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzrfn\" (UniqueName: \"kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.848634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.848476 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.849240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.849190 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.849448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.849423 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.849517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.849499 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.849585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.849562 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.904493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.904422 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzch\" (UniqueName: \"kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:28.905572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.905516 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tqk\" (UniqueName: \"kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:28.951592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.950788 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.951592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.950892 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.951592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.950933 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzrfn\" (UniqueName: \"kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.951592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.951268 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.951592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.951507 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:28.984590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:28.984549 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:21:29.010255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.010225 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:21:29.011645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.011584 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzrfn\" (UniqueName: \"kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:29.052631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.051961 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:21:29.426643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.425897 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn"] Nov 01 14:21:29.433840 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:29.433810 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1782cbaf_0f4d_4c9b_9c0c_e2e3e68e8472.slice/crio-94f565c828bc24c00473900dba05a8fe202866448ea28d91acc14f701d828715 WatchSource:0}: Error finding container 94f565c828bc24c00473900dba05a8fe202866448ea28d91acc14f701d828715: Status 404 returned error can't find the container with id 94f565c828bc24c00473900dba05a8fe202866448ea28d91acc14f701d828715 Nov 01 14:21:29.443479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.443408 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t"] Nov 01 14:21:29.468280 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:29.467437 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cd1543_519f_408c_aacf_b802ca3b49af.slice/crio-91abe94d10345b734d82009f1c4dcf7109c8526df597cbbbcab34752e316bbdc WatchSource:0}: Error finding container 91abe94d10345b734d82009f1c4dcf7109c8526df597cbbbcab34752e316bbdc: Status 404 returned error can't find the container with id 91abe94d10345b734d82009f1c4dcf7109c8526df597cbbbcab34752e316bbdc Nov 01 14:21:29.474137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.474061 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7"] Nov 01 14:21:29.482242 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:29.482197 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55458d1_8174_419d_be24_98611c63cd7e.slice/crio-f594c37a05d13681282385fb64b74ada74af495e14346b2b082c67b5615c57be WatchSource:0}: Error finding container f594c37a05d13681282385fb64b74ada74af495e14346b2b082c67b5615c57be: Status 404 returned error can't find the container with id f594c37a05d13681282385fb64b74ada74af495e14346b2b082c67b5615c57be Nov 01 14:21:29.975723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.975685 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" event={"ID":"d55458d1-8174-419d-be24-98611c63cd7e","Type":"ContainerStarted","Data":"f594c37a05d13681282385fb64b74ada74af495e14346b2b082c67b5615c57be"} Nov 01 14:21:29.977305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.977242 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" event={"ID":"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472","Type":"ContainerStarted","Data":"94f565c828bc24c00473900dba05a8fe202866448ea28d91acc14f701d828715"} Nov 01 14:21:29.978684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:29.978659 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" event={"ID":"f6cd1543-519f-408c-aacf-b802ca3b49af","Type":"ContainerStarted","Data":"91abe94d10345b734d82009f1c4dcf7109c8526df597cbbbcab34752e316bbdc"} Nov 01 14:21:31.126254 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:21:31.126141 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:21:31.511052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:31.510969 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:36.054067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:36.053963 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" event={"ID":"19a80f7f-10df-4f13-9c40-9307098830ab","Type":"ContainerStarted","Data":"3608145129d919fd1c7147721c795695ca7d8ec0492db767fa9a574febdad0cd"} Nov 01 14:21:37.065744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:37.065708 5473 generic.go:356] "Generic (PLEG): container finished" podID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerID="a67575dd6bdde7b363356081a9b5f7c6489708f926cc0c86f026a25f20b4df91" exitCode=0 Nov 01 14:21:37.066161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:37.065822 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" event={"ID":"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0","Type":"ContainerDied","Data":"a67575dd6bdde7b363356081a9b5f7c6489708f926cc0c86f026a25f20b4df91"} Nov 01 14:21:37.073634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:37.073607 5473 generic.go:356] "Generic (PLEG): container finished" podID="19a80f7f-10df-4f13-9c40-9307098830ab" containerID="3608145129d919fd1c7147721c795695ca7d8ec0492db767fa9a574febdad0cd" exitCode=0 Nov 01 14:21:37.073759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:37.073725 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" event={"ID":"19a80f7f-10df-4f13-9c40-9307098830ab","Type":"ContainerDied","Data":"3608145129d919fd1c7147721c795695ca7d8ec0492db767fa9a574febdad0cd"} Nov 01 14:21:38.083238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:38.083192 5473 generic.go:356] "Generic (PLEG): container finished" podID="70162213-073f-4956-a023-023ac5e753e3" containerID="e7e5096a7e1b8571c7a7fcd067acdbb578b850c5490c6260bb593852bcab1a0c" exitCode=0 Nov 01 14:21:38.083718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:38.083355 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" event={"ID":"70162213-073f-4956-a023-023ac5e753e3","Type":"ContainerDied","Data":"e7e5096a7e1b8571c7a7fcd067acdbb578b850c5490c6260bb593852bcab1a0c"} Nov 01 14:21:38.087998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:38.087970 5473 generic.go:356] "Generic (PLEG): container finished" podID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerID="ad7162327634f9fc640c5ed38ec52ac45c6628b9ec80876f6b4c5b765edbca0a" exitCode=0 Nov 01 14:21:38.088097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:38.088017 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" event={"ID":"938fed11-b55f-4b16-a1d2-a54382b86bfd","Type":"ContainerDied","Data":"ad7162327634f9fc640c5ed38ec52ac45c6628b9ec80876f6b4c5b765edbca0a"} Nov 01 14:21:39.094591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:39.094551 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" event={"ID":"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472","Type":"ContainerStarted","Data":"fd9415399466d04c1bc1e73f74e4525e2c2129db12510cf5ba09a943d9c9cfd1"} Nov 01 14:21:39.096305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:39.096271 5473 generic.go:356] "Generic (PLEG): container finished" podID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerID="d4d139ab38792cb899efd47f406c7708324c5d82899c301013b33529eaafef34" exitCode=0 Nov 01 14:21:39.096443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:39.096320 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" event={"ID":"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d","Type":"ContainerDied","Data":"d4d139ab38792cb899efd47f406c7708324c5d82899c301013b33529eaafef34"} Nov 01 14:21:40.101937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.101898 5473 generic.go:356] "Generic (PLEG): container finished" podID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerID="a2cc0228d0de3647110257aab801eabf96aab76ed8f01e90025051e87a64794d" exitCode=0 Nov 01 14:21:40.102422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.101975 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" event={"ID":"35f87c7b-f7af-4eac-b236-ed27e59a6922","Type":"ContainerDied","Data":"a2cc0228d0de3647110257aab801eabf96aab76ed8f01e90025051e87a64794d"} Nov 01 14:21:40.103488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.103468 5473 generic.go:356] "Generic (PLEG): container finished" podID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerID="f49543e4f04cd4ca8e35c8ad1fb5ae6e35685021d86fc03cfa53a7fbbb1574c1" exitCode=0 Nov 01 14:21:40.103578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.103543 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" event={"ID":"f6cd1543-519f-408c-aacf-b802ca3b49af","Type":"ContainerDied","Data":"f49543e4f04cd4ca8e35c8ad1fb5ae6e35685021d86fc03cfa53a7fbbb1574c1"} Nov 01 14:21:40.105132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.105107 5473 generic.go:356] "Generic (PLEG): container finished" podID="d55458d1-8174-419d-be24-98611c63cd7e" containerID="11e1773314369f9b7f8f386b125f51606b42445ac2e371631db3e8b5bea10783" exitCode=0 Nov 01 14:21:40.105230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.105179 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" event={"ID":"d55458d1-8174-419d-be24-98611c63cd7e","Type":"ContainerDied","Data":"11e1773314369f9b7f8f386b125f51606b42445ac2e371631db3e8b5bea10783"} Nov 01 14:21:40.106492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.106470 5473 generic.go:356] "Generic (PLEG): container finished" podID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerID="fd9415399466d04c1bc1e73f74e4525e2c2129db12510cf5ba09a943d9c9cfd1" exitCode=0 Nov 01 14:21:40.106571 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.106542 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" event={"ID":"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472","Type":"ContainerDied","Data":"fd9415399466d04c1bc1e73f74e4525e2c2129db12510cf5ba09a943d9c9cfd1"} Nov 01 14:21:40.112739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.112713 5473 generic.go:356] "Generic (PLEG): container finished" podID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerID="d7521d3ff53aac786a4c2c1e9451c0db2c8ab14ca96eb355897de0d9d2ef830e" exitCode=0 Nov 01 14:21:40.112824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.112805 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" event={"ID":"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0","Type":"ContainerDied","Data":"d7521d3ff53aac786a4c2c1e9451c0db2c8ab14ca96eb355897de0d9d2ef830e"} Nov 01 14:21:40.114536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.114515 5473 generic.go:356] "Generic (PLEG): container finished" podID="19a80f7f-10df-4f13-9c40-9307098830ab" containerID="bcd2e08de76cb06dbcae344c6ff57f00ffddbd7a4a466462a76b77380c0be97c" exitCode=0 Nov 01 14:21:40.114608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:40.114547 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" event={"ID":"19a80f7f-10df-4f13-9c40-9307098830ab","Type":"ContainerDied","Data":"bcd2e08de76cb06dbcae344c6ff57f00ffddbd7a4a466462a76b77380c0be97c"} Nov 01 14:21:41.123138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.123095 5473 generic.go:356] "Generic (PLEG): container finished" podID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerID="e4ea976a076d69feadb3577819287e3d0b45307aa552ac6a90cd2c763b258ac0" exitCode=0 Nov 01 14:21:41.123695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.123197 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" event={"ID":"938fed11-b55f-4b16-a1d2-a54382b86bfd","Type":"ContainerDied","Data":"e4ea976a076d69feadb3577819287e3d0b45307aa552ac6a90cd2c763b258ac0"} Nov 01 14:21:41.126994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.126965 5473 generic.go:356] "Generic (PLEG): container finished" podID="d55458d1-8174-419d-be24-98611c63cd7e" containerID="b51849c14ec330d08f1c86aa38e02c788e32aaf0eb5a082f99af4a36e0bb3431" exitCode=0 Nov 01 14:21:41.127100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.127080 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" event={"ID":"d55458d1-8174-419d-be24-98611c63cd7e","Type":"ContainerDied","Data":"b51849c14ec330d08f1c86aa38e02c788e32aaf0eb5a082f99af4a36e0bb3431"} Nov 01 14:21:41.131483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.131457 5473 generic.go:356] "Generic (PLEG): container finished" podID="70162213-073f-4956-a023-023ac5e753e3" containerID="418d9595943863f45e99809d012d9eb8ce990ae18b50b5abb4003e3ea3b73292" exitCode=0 Nov 01 14:21:41.131609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.131514 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" event={"ID":"70162213-073f-4956-a023-023ac5e753e3","Type":"ContainerDied","Data":"418d9595943863f45e99809d012d9eb8ce990ae18b50b5abb4003e3ea3b73292"} Nov 01 14:21:41.136180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.136139 5473 generic.go:356] "Generic (PLEG): container finished" podID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerID="680e6ab2e1b986116178fcc09ba30858ce6f5229bbc1e86a56e268fb56dc149a" exitCode=0 Nov 01 14:21:41.136295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:41.136181 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" event={"ID":"f6cd1543-519f-408c-aacf-b802ca3b49af","Type":"ContainerDied","Data":"680e6ab2e1b986116178fcc09ba30858ce6f5229bbc1e86a56e268fb56dc149a"} Nov 01 14:21:42.150472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:42.149268 5473 generic.go:356] "Generic (PLEG): container finished" podID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerID="75595fb067a64dceb10bc94dfc5f8f4cd962bde65ba34aa5cea00cdefc6a584f" exitCode=0 Nov 01 14:21:42.150472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:42.149327 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" event={"ID":"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472","Type":"ContainerDied","Data":"75595fb067a64dceb10bc94dfc5f8f4cd962bde65ba34aa5cea00cdefc6a584f"} Nov 01 14:21:43.159603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:43.158451 5473 generic.go:356] "Generic (PLEG): container finished" podID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerID="3990bab9350ebdb76b5d977098d577473e24bbb0c17342ee26c50512e9a4040d" exitCode=0 Nov 01 14:21:43.159603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:43.158549 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" event={"ID":"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d","Type":"ContainerDied","Data":"3990bab9350ebdb76b5d977098d577473e24bbb0c17342ee26c50512e9a4040d"} Nov 01 14:21:43.937195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:43.937117 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:48.205667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:48.204576 5473 generic.go:356] "Generic (PLEG): container finished" podID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerID="8df9f324e798f256e4c5ca6f30a6536c9900720a4b154dd54bd3147eff10195e" exitCode=0 Nov 01 14:21:48.205667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:48.204696 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" event={"ID":"35f87c7b-f7af-4eac-b236-ed27e59a6922","Type":"ContainerDied","Data":"8df9f324e798f256e4c5ca6f30a6536c9900720a4b154dd54bd3147eff10195e"} Nov 01 14:21:52.692596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:52.692561 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:52.856897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:52.856831 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:53.351985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.351931 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:53.554978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.554903 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7"] Nov 01 14:21:53.562623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.562590 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.604298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.604111 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:53.611087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.611054 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b67b16-36e1-4426-9758-8968eef243d3-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.611279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.611131 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdwn6\" (UniqueName: \"kubernetes.io/projected/90b67b16-36e1-4426-9758-8968eef243d3-kube-api-access-rdwn6\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.611279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.611172 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.611279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.611256 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90b67b16-36e1-4426-9758-8968eef243d3-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.611464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.611295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.611464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.611328 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.678333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.678246 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 14:21:53.678580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.678388 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 14:21:53.678580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.678500 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 14:21:53.678719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.678589 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 14:21:53.678719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.678665 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kgzw7\"" Nov 01 14:21:53.678719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.678712 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 14:21:53.685120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.685095 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7"] Nov 01 14:21:53.712329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.712293 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.712829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.712352 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.712829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.712433 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b67b16-36e1-4426-9758-8968eef243d3-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.712829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.712516 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdwn6\" (UniqueName: \"kubernetes.io/projected/90b67b16-36e1-4426-9758-8968eef243d3-kube-api-access-rdwn6\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.712829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.712550 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.712829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.712612 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90b67b16-36e1-4426-9758-8968eef243d3-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.713125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.713103 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/90b67b16-36e1-4426-9758-8968eef243d3-manifests\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.724687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.724230 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.724956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.724898 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90b67b16-36e1-4426-9758-8968eef243d3-tmp\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.725620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.725575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-metrics-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.730590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.730544 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/90b67b16-36e1-4426-9758-8968eef243d3-webhook-cert\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.771950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.771880 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdwn6\" (UniqueName: \"kubernetes.io/projected/90b67b16-36e1-4426-9758-8968eef243d3-kube-api-access-rdwn6\") pod \"opendatahub-operator-controller-manager-666cc8ff84-hjvf7\" (UID: \"90b67b16-36e1-4426-9758-8968eef243d3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.788854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.788765 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:53.887724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.887338 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:21:53.990281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:53.990235 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:54.274316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:54.274281 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:54.437788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:54.437511 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7"] Nov 01 14:21:54.446727 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:21:54.446682 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b67b16_36e1_4426_9758_8968eef243d3.slice/crio-6d40f7b9cb580b9bf5bbb59c8b9721ab71a15fcfff71c8c5c0bf9bde1d6b9398 WatchSource:0}: Error finding container 6d40f7b9cb580b9bf5bbb59c8b9721ab71a15fcfff71c8c5c0bf9bde1d6b9398: Status 404 returned error can't find the container with id 6d40f7b9cb580b9bf5bbb59c8b9721ab71a15fcfff71c8c5c0bf9bde1d6b9398 Nov 01 14:21:55.284626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:55.284541 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" event={"ID":"90b67b16-36e1-4426-9758-8968eef243d3","Type":"ContainerStarted","Data":"6d40f7b9cb580b9bf5bbb59c8b9721ab71a15fcfff71c8c5c0bf9bde1d6b9398"} Nov 01 14:21:56.311737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:56.311584 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:56.485314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:56.485226 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:21:59.340777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:59.340723 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" event={"ID":"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0","Type":"ContainerStarted","Data":"6db877d733e6b20efc7a4e097a56931475e0bffd77f91dce3517a1c6457c6693"} Nov 01 14:21:59.458236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:21:59.458155 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" podStartSLOduration=2.905420894 podStartE2EDuration="38.458134929s" totalImagesPullingTime="35.552714035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:21:59.457350829 +0000 UTC m=+258.515502676" watchObservedRunningTime="2026-11-01 14:21:59.458134929 +0000 UTC m=+258.516286780" Nov 01 14:22:00.363248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:00.362699 5473 generic.go:356] "Generic (PLEG): container finished" podID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerID="6db877d733e6b20efc7a4e097a56931475e0bffd77f91dce3517a1c6457c6693" exitCode=0 Nov 01 14:22:00.363248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:00.362815 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" event={"ID":"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0","Type":"ContainerDied","Data":"6db877d733e6b20efc7a4e097a56931475e0bffd77f91dce3517a1c6457c6693"} Nov 01 14:22:00.369313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:00.369025 5473 generic.go:356] "Generic (PLEG): container finished" podID="19a80f7f-10df-4f13-9c40-9307098830ab" containerID="d1a7ecc8f7ea97a922b05203ebd6292ec6f39a2d078824bf7e2c5421232123bf" exitCode=0 Nov 01 14:22:00.369313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:00.369107 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" event={"ID":"19a80f7f-10df-4f13-9c40-9307098830ab","Type":"ContainerDied","Data":"d1a7ecc8f7ea97a922b05203ebd6292ec6f39a2d078824bf7e2c5421232123bf"} Nov 01 14:22:00.916312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:00.916282 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:01.078185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.078154 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:01.120003 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:22:01.119967 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:22:01.384098 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.383737 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" event={"ID":"d55458d1-8174-419d-be24-98611c63cd7e","Type":"ContainerStarted","Data":"77164d17ea6d04b42008af188e5a651c60a9fc373d90650887b76b0441281621"} Nov 01 14:22:01.500330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.499189 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" podStartSLOduration=3.722200889 podStartE2EDuration="33.499170949s" totalImagesPullingTime="29.77697006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:01.498701825 +0000 UTC m=+260.556853686" watchObservedRunningTime="2026-11-01 14:22:01.499170949 +0000 UTC m=+260.557322800" Nov 01 14:22:01.580503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.580452 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:22:01.636367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.636337 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:22:01.702784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.702750 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22\" (UniqueName: \"kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22\") pod \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " Nov 01 14:22:01.702920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.702842 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle\") pod \"19a80f7f-10df-4f13-9c40-9307098830ab\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " Nov 01 14:22:01.702920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.702881 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz\" (UniqueName: \"kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz\") pod \"19a80f7f-10df-4f13-9c40-9307098830ab\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " Nov 01 14:22:01.702920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.702907 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util\") pod \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " Nov 01 14:22:01.703035 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.702962 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle\") pod \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\" (UID: \"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0\") " Nov 01 14:22:01.703068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.703040 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util\") pod \"19a80f7f-10df-4f13-9c40-9307098830ab\" (UID: \"19a80f7f-10df-4f13-9c40-9307098830ab\") " Nov 01 14:22:01.710677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.710615 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz" pod "19a80f7f-10df-4f13-9c40-9307098830ab" (UID: "19a80f7f-10df-4f13-9c40-9307098830ab"). InnerVolumeSpecName "kube-api-access-trmtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:01.710913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.710883 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util" pod "19a80f7f-10df-4f13-9c40-9307098830ab" (UID: "19a80f7f-10df-4f13-9c40-9307098830ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:01.711133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.711115 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22" pod "42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" (UID: "42a3003d-f42d-40a6-bd2c-bfdcc254b8b0"). InnerVolumeSpecName "kube-api-access-wxh22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:01.711783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.711752 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle" pod "42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" (UID: "42a3003d-f42d-40a6-bd2c-bfdcc254b8b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:01.712364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.712330 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle" pod "19a80f7f-10df-4f13-9c40-9307098830ab" (UID: "19a80f7f-10df-4f13-9c40-9307098830ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:01.719313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.719283 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util" pod "42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" (UID: "42a3003d-f42d-40a6-bd2c-bfdcc254b8b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:01.804736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.804701 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:01.804736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.804738 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:01.804924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.804755 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxh22\" (UniqueName: \"kubernetes.io/projected/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-kube-api-access-wxh22\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:01.804924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.804771 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/19a80f7f-10df-4f13-9c40-9307098830ab-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:01.804924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.804786 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmtz\" (UniqueName: \"kubernetes.io/projected/19a80f7f-10df-4f13-9c40-9307098830ab-kube-api-access-trmtz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:01.804924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:01.804799 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42a3003d-f42d-40a6-bd2c-bfdcc254b8b0-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:02.396446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.396408 5473 generic.go:356] "Generic (PLEG): container finished" podID="70162213-073f-4956-a023-023ac5e753e3" containerID="b6ebcd25904529cd9549a233c8b08aa9c00999b34c17ea9e435d06563c66927b" exitCode=0 Nov 01 14:22:02.396879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.396493 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" event={"ID":"70162213-073f-4956-a023-023ac5e753e3","Type":"ContainerDied","Data":"b6ebcd25904529cd9549a233c8b08aa9c00999b34c17ea9e435d06563c66927b"} Nov 01 14:22:02.399084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.399051 5473 generic.go:356] "Generic (PLEG): container finished" podID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerID="1de9298de1a8035d122d202b7a63d3827d734cc470663493665b1c9118b3ea2c" exitCode=0 Nov 01 14:22:02.399231 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.399129 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" event={"ID":"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472","Type":"ContainerDied","Data":"1de9298de1a8035d122d202b7a63d3827d734cc470663493665b1c9118b3ea2c"} Nov 01 14:22:02.401406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.401378 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" event={"ID":"42a3003d-f42d-40a6-bd2c-bfdcc254b8b0","Type":"ContainerDied","Data":"4d2cf288e2ed7d63158ab2a3606ca115167cce286bc50718ccf193eafc9b0deb"} Nov 01 14:22:02.401523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.401425 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d2cf288e2ed7d63158ab2a3606ca115167cce286bc50718ccf193eafc9b0deb" Nov 01 14:22:02.401523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.401385 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axl26t" Nov 01 14:22:02.404570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.404546 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" Nov 01 14:22:02.404686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.404545 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nhslp" event={"ID":"19a80f7f-10df-4f13-9c40-9307098830ab","Type":"ContainerDied","Data":"b41e8fa0e1c36ca66a014f2246e329f0394de9043d8ec0d57e89b98e72d16bca"} Nov 01 14:22:02.404686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.404656 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b41e8fa0e1c36ca66a014f2246e329f0394de9043d8ec0d57e89b98e72d16bca" Nov 01 14:22:02.412618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.412494 5473 generic.go:356] "Generic (PLEG): container finished" podID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerID="f567be7f9ea7b572166ab434fd81a156bdf583bbf4c5837207f45d29e0458a33" exitCode=0 Nov 01 14:22:02.412618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.412573 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" event={"ID":"35f87c7b-f7af-4eac-b236-ed27e59a6922","Type":"ContainerDied","Data":"f567be7f9ea7b572166ab434fd81a156bdf583bbf4c5837207f45d29e0458a33"} Nov 01 14:22:02.415984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.415959 5473 generic.go:356] "Generic (PLEG): container finished" podID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerID="04ce49c2934c4483c54eb22441792ed82b1dd87db8976c09b3a2d58f9f2b1cb7" exitCode=0 Nov 01 14:22:02.416126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.415989 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" event={"ID":"f6cd1543-519f-408c-aacf-b802ca3b49af","Type":"ContainerDied","Data":"04ce49c2934c4483c54eb22441792ed82b1dd87db8976c09b3a2d58f9f2b1cb7"} Nov 01 14:22:02.418903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.418881 5473 generic.go:356] "Generic (PLEG): container finished" podID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerID="858f79c6fe1e3b19b0c32725d4c99bf825a9ee5a3393f4b1b1b470be3af7007d" exitCode=0 Nov 01 14:22:02.418989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.418913 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" event={"ID":"938fed11-b55f-4b16-a1d2-a54382b86bfd","Type":"ContainerDied","Data":"858f79c6fe1e3b19b0c32725d4c99bf825a9ee5a3393f4b1b1b470be3af7007d"} Nov 01 14:22:02.421938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.421919 5473 generic.go:356] "Generic (PLEG): container finished" podID="d55458d1-8174-419d-be24-98611c63cd7e" containerID="77164d17ea6d04b42008af188e5a651c60a9fc373d90650887b76b0441281621" exitCode=0 Nov 01 14:22:02.422041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:02.421946 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" event={"ID":"d55458d1-8174-419d-be24-98611c63cd7e","Type":"ContainerDied","Data":"77164d17ea6d04b42008af188e5a651c60a9fc373d90650887b76b0441281621"} Nov 01 14:22:03.428145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.428055 5473 generic.go:356] "Generic (PLEG): container finished" podID="90b67b16-36e1-4426-9758-8968eef243d3" containerID="00719da3e4c3f1a91c61d159bfd77fdb48b4ef14a9fa2ea3a78d6288557343eb" exitCode=0 Nov 01 14:22:03.428629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.428136 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" event={"ID":"90b67b16-36e1-4426-9758-8968eef243d3","Type":"ContainerDied","Data":"00719da3e4c3f1a91c61d159bfd77fdb48b4ef14a9fa2ea3a78d6288557343eb"} Nov 01 14:22:03.430378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.430361 5473 generic.go:356] "Generic (PLEG): container finished" podID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerID="541d9a683d24884c53e12bd7ab53dcf7cecd30ab5fbaaa20cb7a129ffed6715f" exitCode=0 Nov 01 14:22:03.430466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.430414 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" event={"ID":"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d","Type":"ContainerDied","Data":"541d9a683d24884c53e12bd7ab53dcf7cecd30ab5fbaaa20cb7a129ffed6715f"} Nov 01 14:22:03.588512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.588490 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:22:03.645948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.645924 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:22:03.669817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.669794 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:22:03.672991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.672961 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:22:03.676415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.676397 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:22:03.679490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.679449 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:22:03.720160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720131 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle\") pod \"35f87c7b-f7af-4eac-b236-ed27e59a6922\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " Nov 01 14:22:03.720329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720174 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn\" (UniqueName: \"kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn\") pod \"d55458d1-8174-419d-be24-98611c63cd7e\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " Nov 01 14:22:03.720329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720209 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util\") pod \"35f87c7b-f7af-4eac-b236-ed27e59a6922\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " Nov 01 14:22:03.720329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720262 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util\") pod \"f6cd1543-519f-408c-aacf-b802ca3b49af\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " Nov 01 14:22:03.720329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720298 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk\" (UniqueName: \"kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk\") pod \"f6cd1543-519f-408c-aacf-b802ca3b49af\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " Nov 01 14:22:03.720552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720341 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util\") pod \"70162213-073f-4956-a023-023ac5e753e3\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " Nov 01 14:22:03.720552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720387 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util\") pod \"938fed11-b55f-4b16-a1d2-a54382b86bfd\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " Nov 01 14:22:03.720552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720422 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle\") pod \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " Nov 01 14:22:03.720552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720469 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch\" (UniqueName: \"kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch\") pod \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " Nov 01 14:22:03.720552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720497 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util\") pod \"d55458d1-8174-419d-be24-98611c63cd7e\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " Nov 01 14:22:03.720552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720532 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle\") pod \"938fed11-b55f-4b16-a1d2-a54382b86bfd\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720561 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg\" (UniqueName: \"kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg\") pod \"938fed11-b55f-4b16-a1d2-a54382b86bfd\" (UID: \"938fed11-b55f-4b16-a1d2-a54382b86bfd\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720586 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle" pod "35f87c7b-f7af-4eac-b236-ed27e59a6922" (UID: "35f87c7b-f7af-4eac-b236-ed27e59a6922"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720591 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx\" (UniqueName: \"kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx\") pod \"35f87c7b-f7af-4eac-b236-ed27e59a6922\" (UID: \"35f87c7b-f7af-4eac-b236-ed27e59a6922\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720654 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle\") pod \"d55458d1-8174-419d-be24-98611c63cd7e\" (UID: \"d55458d1-8174-419d-be24-98611c63cd7e\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720700 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc\" (UniqueName: \"kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc\") pod \"70162213-073f-4956-a023-023ac5e753e3\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720731 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util\") pod \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\" (UID: \"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720777 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle\") pod \"70162213-073f-4956-a023-023ac5e753e3\" (UID: \"70162213-073f-4956-a023-023ac5e753e3\") " Nov 01 14:22:03.720847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.720812 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle\") pod \"f6cd1543-519f-408c-aacf-b802ca3b49af\" (UID: \"f6cd1543-519f-408c-aacf-b802ca3b49af\") " Nov 01 14:22:03.721257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.721104 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.721860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.721840 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle" pod "f6cd1543-519f-408c-aacf-b802ca3b49af" (UID: "f6cd1543-519f-408c-aacf-b802ca3b49af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.722559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.722526 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle" pod "938fed11-b55f-4b16-a1d2-a54382b86bfd" (UID: "938fed11-b55f-4b16-a1d2-a54382b86bfd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.722829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.722661 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle" pod "d55458d1-8174-419d-be24-98611c63cd7e" (UID: "d55458d1-8174-419d-be24-98611c63cd7e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.723055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.723023 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx" pod "35f87c7b-f7af-4eac-b236-ed27e59a6922" (UID: "35f87c7b-f7af-4eac-b236-ed27e59a6922"). InnerVolumeSpecName "kube-api-access-ssddx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:03.724171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.723462 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle" pod "70162213-073f-4956-a023-023ac5e753e3" (UID: "70162213-073f-4956-a023-023ac5e753e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.724171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.723938 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle" pod "1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" (UID: "1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.724337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.724199 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch" pod "1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" (UID: "1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472"). InnerVolumeSpecName "kube-api-access-cfzch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:03.725759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.725735 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn" pod "d55458d1-8174-419d-be24-98611c63cd7e" (UID: "d55458d1-8174-419d-be24-98611c63cd7e"). InnerVolumeSpecName "kube-api-access-wzrfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:03.727788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.727766 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util" pod "f6cd1543-519f-408c-aacf-b802ca3b49af" (UID: "f6cd1543-519f-408c-aacf-b802ca3b49af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.728758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.728735 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util" pod "1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" (UID: "1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.728827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.728741 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util" pod "35f87c7b-f7af-4eac-b236-ed27e59a6922" (UID: "35f87c7b-f7af-4eac-b236-ed27e59a6922"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.729197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.729175 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg" pod "938fed11-b55f-4b16-a1d2-a54382b86bfd" (UID: "938fed11-b55f-4b16-a1d2-a54382b86bfd"). InnerVolumeSpecName "kube-api-access-kscdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:03.729197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.729185 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk" pod "f6cd1543-519f-408c-aacf-b802ca3b49af" (UID: "f6cd1543-519f-408c-aacf-b802ca3b49af"). InnerVolumeSpecName "kube-api-access-x6tqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:03.730300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.730278 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc" pod "70162213-073f-4956-a023-023ac5e753e3" (UID: "70162213-073f-4956-a023-023ac5e753e3"). InnerVolumeSpecName "kube-api-access-fkmjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:03.730936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.730914 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util" pod "d55458d1-8174-419d-be24-98611c63cd7e" (UID: "d55458d1-8174-419d-be24-98611c63cd7e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.730971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.730958 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util" pod "70162213-073f-4956-a023-023ac5e753e3" (UID: "70162213-073f-4956-a023-023ac5e753e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.732019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.732000 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util" pod "938fed11-b55f-4b16-a1d2-a54382b86bfd" (UID: "938fed11-b55f-4b16-a1d2-a54382b86bfd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:03.822189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822155 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzrfn\" (UniqueName: \"kubernetes.io/projected/d55458d1-8174-419d-be24-98611c63cd7e-kube-api-access-wzrfn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822183 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/35f87c7b-f7af-4eac-b236-ed27e59a6922-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822192 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822202 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tqk\" (UniqueName: \"kubernetes.io/projected/f6cd1543-519f-408c-aacf-b802ca3b49af-kube-api-access-x6tqk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822231 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822245 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822257 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822269 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfzch\" (UniqueName: \"kubernetes.io/projected/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-kube-api-access-cfzch\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822279 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822287 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/938fed11-b55f-4b16-a1d2-a54382b86bfd-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822295 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kscdg\" (UniqueName: \"kubernetes.io/projected/938fed11-b55f-4b16-a1d2-a54382b86bfd-kube-api-access-kscdg\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822303 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssddx\" (UniqueName: \"kubernetes.io/projected/35f87c7b-f7af-4eac-b236-ed27e59a6922-kube-api-access-ssddx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822313 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d55458d1-8174-419d-be24-98611c63cd7e-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822321 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmjc\" (UniqueName: \"kubernetes.io/projected/70162213-073f-4956-a023-023ac5e753e3-kube-api-access-fkmjc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822329 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822336 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70162213-073f-4956-a023-023ac5e753e3-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:03.822484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:03.822345 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f6cd1543-519f-408c-aacf-b802ca3b49af-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:04.437058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.437033 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" Nov 01 14:22:04.437058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.437032 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69p7f8" event={"ID":"938fed11-b55f-4b16-a1d2-a54382b86bfd","Type":"ContainerDied","Data":"5f7b483f27bae7e45175fc287eb53e5dfdd1648d819ec20cef2ad1c78979eb4a"} Nov 01 14:22:04.437593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.437082 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7b483f27bae7e45175fc287eb53e5dfdd1648d819ec20cef2ad1c78979eb4a" Nov 01 14:22:04.439040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.439020 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" Nov 01 14:22:04.439170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.439035 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74dxz7" event={"ID":"d55458d1-8174-419d-be24-98611c63cd7e","Type":"ContainerDied","Data":"f594c37a05d13681282385fb64b74ada74af495e14346b2b082c67b5615c57be"} Nov 01 14:22:04.439170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.439077 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f594c37a05d13681282385fb64b74ada74af495e14346b2b082c67b5615c57be" Nov 01 14:22:04.440919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.440895 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" event={"ID":"70162213-073f-4956-a023-023ac5e753e3","Type":"ContainerDied","Data":"99aac05f7658341fbb8224f5bbecef17669cccb9fcf1d23a6bfa3f71d227c889"} Nov 01 14:22:04.440919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.440912 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czpmhr" Nov 01 14:22:04.441045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.440942 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99aac05f7658341fbb8224f5bbecef17669cccb9fcf1d23a6bfa3f71d227c889" Nov 01 14:22:04.443126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.443106 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" Nov 01 14:22:04.443248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.443125 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6cwvcn" event={"ID":"1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472","Type":"ContainerDied","Data":"94f565c828bc24c00473900dba05a8fe202866448ea28d91acc14f701d828715"} Nov 01 14:22:04.443248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.443166 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f565c828bc24c00473900dba05a8fe202866448ea28d91acc14f701d828715" Nov 01 14:22:04.445524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.445507 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" Nov 01 14:22:04.445524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.445511 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644r5m79" event={"ID":"35f87c7b-f7af-4eac-b236-ed27e59a6922","Type":"ContainerDied","Data":"b91272d088e6521bbb422ca674e30f8070dd5083e1482957d89938b9b3b5b3ac"} Nov 01 14:22:04.445656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.445547 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b91272d088e6521bbb422ca674e30f8070dd5083e1482957d89938b9b3b5b3ac" Nov 01 14:22:04.448191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.448071 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" event={"ID":"f6cd1543-519f-408c-aacf-b802ca3b49af","Type":"ContainerDied","Data":"91abe94d10345b734d82009f1c4dcf7109c8526df597cbbbcab34752e316bbdc"} Nov 01 14:22:04.448191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.448104 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91abe94d10345b734d82009f1c4dcf7109c8526df597cbbbcab34752e316bbdc" Nov 01 14:22:04.448191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.448178 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ak4r5t" Nov 01 14:22:04.451716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.451683 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" event={"ID":"90b67b16-36e1-4426-9758-8968eef243d3","Type":"ContainerStarted","Data":"b09f79ba42ba199af53e18f66d559dbded8dd9e8f6fd1748e78e29f3bf93b8e5"} Nov 01 14:22:04.452432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.452417 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:22:04.564499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.564475 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:22:04.579841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.579788 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" podStartSLOduration=2.071128931 podStartE2EDuration="11.579770942s" totalImagesPullingTime="9.011526777s" totalInitContainerRuntime="497.115234ms" isStatefulPod=false podCreationTimestamp="2026-11-01 14:21:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:04.578190667 +0000 UTC m=+263.636342517" watchObservedRunningTime="2026-11-01 14:22:04.579770942 +0000 UTC m=+263.637922793" Nov 01 14:22:04.630036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.630005 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util\") pod \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " Nov 01 14:22:04.630267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.630063 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g\" (UniqueName: \"kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g\") pod \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " Nov 01 14:22:04.630267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.630124 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle\") pod \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\" (UID: \"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d\") " Nov 01 14:22:04.630901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.630870 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle" pod "1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" (UID: "1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:04.632350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.632322 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g" pod "1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" (UID: "1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d"). InnerVolumeSpecName "kube-api-access-n8t8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:22:04.635640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.635616 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util" pod "1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" (UID: "1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:22:04.731452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.731414 5473 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-util\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:04.731452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.731443 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8t8g\" (UniqueName: \"kubernetes.io/projected/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-kube-api-access-n8t8g\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:04.731452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:04.731454 5473 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d-bundle\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:22:05.458777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:05.458689 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" event={"ID":"1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d","Type":"ContainerDied","Data":"f5432f0041c51f504b12019f8bc23a73c6a62a9a2c2ff556e188717b6e8cbf4e"} Nov 01 14:22:05.458777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:05.458724 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cg42lm" Nov 01 14:22:05.458777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:05.458748 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5432f0041c51f504b12019f8bc23a73c6a62a9a2c2ff556e188717b6e8cbf4e" Nov 01 14:22:10.082907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:10.082871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:11.690261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:11.690227 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:13.771097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771056 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k"] Nov 01 14:22:13.771612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771546 5473 cpu_manager.go:414] "containerMap: removing container" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="extract" Nov 01 14:22:13.771612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771564 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="extract" Nov 01 14:22:13.771612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771586 5473 cpu_manager.go:414] "containerMap: removing container" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="util" Nov 01 14:22:13.771612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771593 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="util" Nov 01 14:22:13.771612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771602 5473 cpu_manager.go:414] "containerMap: removing container" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="extract" Nov 01 14:22:13.771612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771609 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771623 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771631 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771640 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771647 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771662 5473 cpu_manager.go:414] "containerMap: removing container" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771669 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771677 5473 cpu_manager.go:414] "containerMap: removing container" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771685 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771695 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771702 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771718 5473 cpu_manager.go:414] "containerMap: removing container" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771725 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771735 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771742 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771753 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771760 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771768 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771775 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771785 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771793 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771808 5473 cpu_manager.go:414] "containerMap: removing container" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771815 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771824 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771831 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771847 5473 cpu_manager.go:414] "containerMap: removing container" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771854 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771873 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771879 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771892 5473 cpu_manager.go:414] "containerMap: removing container" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771899 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="util" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771909 5473 cpu_manager.go:414] "containerMap: removing container" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771916 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771926 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771934 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="extract" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771945 5473 cpu_manager.go:414] "containerMap: removing container" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771952 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="pull" Nov 01 14:22:13.771946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771961 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="pull" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771968 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="pull" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771977 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771984 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.771997 5473 cpu_manager.go:414] "containerMap: removing container" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772004 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772013 5473 cpu_manager.go:414] "containerMap: removing container" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772020 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772030 5473 cpu_manager.go:414] "containerMap: removing container" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772038 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772052 5473 cpu_manager.go:414] "containerMap: removing container" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="pull" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772059 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="pull" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772134 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55458d1-8174-419d-be24-98611c63cd7e" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772149 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9f2d61-35fd-4fe4-9db1-d320c0f8de8d" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772159 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6cd1543-519f-408c-aacf-b802ca3b49af" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772168 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a80f7f-10df-4f13-9c40-9307098830ab" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772177 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="938fed11-b55f-4b16-a1d2-a54382b86bfd" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772185 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="70162213-073f-4956-a023-023ac5e753e3" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772194 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f87c7b-f7af-4eac-b236-ed27e59a6922" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772206 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a3003d-f42d-40a6-bd2c-bfdcc254b8b0" containerName="extract" Nov 01 14:22:13.773512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.772230 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1782cbaf-0f4d-4c9b-9c0c-e2e3e68e8472" containerName="extract" Nov 01 14:22:13.778748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.778724 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:13.827143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.827107 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k"] Nov 01 14:22:13.875731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.875699 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 14:22:13.875731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.875699 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 14:22:13.875936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.875818 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-57g78\"" Nov 01 14:22:13.907317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:13.907284 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmc5n\" (UniqueName: \"kubernetes.io/projected/2a92488d-7096-4b4c-ab4f-79bf4dd16e94-kube-api-access-rmc5n\") pod \"limitador-operator-controller-manager-5778bb4449-6gn9k\" (UID: \"2a92488d-7096-4b4c-ab4f-79bf4dd16e94\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:14.007954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:14.007915 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmc5n\" (UniqueName: \"kubernetes.io/projected/2a92488d-7096-4b4c-ab4f-79bf4dd16e94-kube-api-access-rmc5n\") pod \"limitador-operator-controller-manager-5778bb4449-6gn9k\" (UID: \"2a92488d-7096-4b4c-ab4f-79bf4dd16e94\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:14.069667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:14.069600 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmc5n\" (UniqueName: \"kubernetes.io/projected/2a92488d-7096-4b4c-ab4f-79bf4dd16e94-kube-api-access-rmc5n\") pod \"limitador-operator-controller-manager-5778bb4449-6gn9k\" (UID: \"2a92488d-7096-4b4c-ab4f-79bf4dd16e94\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:14.091421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:14.091397 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:14.456629 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:22:14.456595 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a92488d_7096_4b4c_ab4f_79bf4dd16e94.slice/crio-df0a74e180fd59e2b848543afba202cb01089214a91267437d9e9f340bb7500a WatchSource:0}: Error finding container df0a74e180fd59e2b848543afba202cb01089214a91267437d9e9f340bb7500a: Status 404 returned error can't find the container with id df0a74e180fd59e2b848543afba202cb01089214a91267437d9e9f340bb7500a Nov 01 14:22:14.458782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:14.458591 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k"] Nov 01 14:22:14.495235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:14.495160 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:14.498999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:14.498972 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" event={"ID":"2a92488d-7096-4b4c-ab4f-79bf4dd16e94","Type":"ContainerStarted","Data":"df0a74e180fd59e2b848543afba202cb01089214a91267437d9e9f340bb7500a"} Nov 01 14:22:15.336229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:15.336167 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:16.132521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.132490 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d"] Nov 01 14:22:16.137165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.137144 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.176364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.176326 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d"] Nov 01 14:22:16.224985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.224949 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 14:22:16.224985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.224986 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-w9brc\"" Nov 01 14:22:16.225184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.224986 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 14:22:16.229334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.229312 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dmc\" (UniqueName: \"kubernetes.io/projected/9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be-kube-api-access-b7dmc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d\" (UID: \"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.229491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.229433 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d\" (UID: \"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.330071 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.330038 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d\" (UID: \"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.330278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.330090 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dmc\" (UniqueName: \"kubernetes.io/projected/9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be-kube-api-access-b7dmc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d\" (UID: \"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.330535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.330503 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d\" (UID: \"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.387001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.386922 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dmc\" (UniqueName: \"kubernetes.io/projected/9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be-kube-api-access-b7dmc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d\" (UID: \"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.448942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.448905 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" Nov 01 14:22:16.466334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.466311 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-666cc8ff84-hjvf7" Nov 01 14:22:16.878871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:16.878730 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d"] Nov 01 14:22:16.878871 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:22:16.878872 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3c1fc0_ab9c_4e2e_b758_0bab1eeec8be.slice/crio-e26edb919962b81d86403dc8677ead32349a0beaeac951a37cbd7159653c5c7b WatchSource:0}: Error finding container e26edb919962b81d86403dc8677ead32349a0beaeac951a37cbd7159653c5c7b: Status 404 returned error can't find the container with id e26edb919962b81d86403dc8677ead32349a0beaeac951a37cbd7159653c5c7b Nov 01 14:22:17.349841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:17.349785 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:17.516755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:17.516719 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" event={"ID":"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be","Type":"ContainerStarted","Data":"e26edb919962b81d86403dc8677ead32349a0beaeac951a37cbd7159653c5c7b"} Nov 01 14:22:19.489023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:19.488977 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:19.528776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:19.528733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" event={"ID":"2a92488d-7096-4b4c-ab4f-79bf4dd16e94","Type":"ContainerStarted","Data":"7e831ea8d68e2fdc0fb35af60a18f49af254346ee1f18d7e663cc7f26a632cf1"} Nov 01 14:22:19.528968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:19.528832 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:19.644190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:19.644110 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" podStartSLOduration=2.4474248100000002 podStartE2EDuration="6.644071406s" totalImagesPullingTime="4.196646596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:19.643161749 +0000 UTC m=+278.701313598" watchObservedRunningTime="2026-11-01 14:22:19.644071406 +0000 UTC m=+278.702223258" Nov 01 14:22:20.299709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:20.299674 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:20.535968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:20.535926 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" event={"ID":"9c3c1fc0-ab9c-4e2e-b758-0bab1eeec8be","Type":"ContainerStarted","Data":"90f0b68e3c6d1998c96f527a8a0b7a66bcfb9d139de15fd02b6769b3162e9d5a"} Nov 01 14:22:20.754006 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:20.753905 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d6w9d" podStartSLOduration=1.989051782 podStartE2EDuration="4.753889058s" totalImagesPullingTime="2.764837276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:20.752748687 +0000 UTC m=+279.810900537" watchObservedRunningTime="2026-11-01 14:22:20.753889058 +0000 UTC m=+279.812040909" Nov 01 14:22:20.939795 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:20.939724 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:21.207941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:21.207908 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:21.452469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:21.452400 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:21.665199 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:21.665136 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:21.871485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:21.871415 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:22.382530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.382496 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-phggr"] Nov 01 14:22:22.385543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.385526 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.445203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.445178 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-phggr"] Nov 01 14:22:22.487136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.487106 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-phggr\" (UID: \"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.487302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.487144 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7lm\" (UniqueName: \"kubernetes.io/projected/58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9-kube-api-access-db7lm\") pod \"cert-manager-webhook-5dbbbf75cc-phggr\" (UID: \"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.493582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.493563 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 14:22:22.493658 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.493602 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 14:22:22.543878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.543844 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nwcs4\"" Nov 01 14:22:22.588206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.588168 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-phggr\" (UID: \"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.588206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.588211 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7lm\" (UniqueName: \"kubernetes.io/projected/58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9-kube-api-access-db7lm\") pod \"cert-manager-webhook-5dbbbf75cc-phggr\" (UID: \"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.652684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.652609 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-phggr\" (UID: \"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.664810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.664777 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7lm\" (UniqueName: \"kubernetes.io/projected/58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9-kube-api-access-db7lm\") pod \"cert-manager-webhook-5dbbbf75cc-phggr\" (UID: \"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.707838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.707800 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:22.754699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:22.754657 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:23.070708 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:22:23.070677 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58a5c2fa_1637_4f72_9ad2_1b6a7f9107b9.slice/crio-fae71f20266a7700ca5bf301e22f16a067e0a3d31c5bf8173d6fdf8387e39c60 WatchSource:0}: Error finding container fae71f20266a7700ca5bf301e22f16a067e0a3d31c5bf8173d6fdf8387e39c60: Status 404 returned error can't find the container with id fae71f20266a7700ca5bf301e22f16a067e0a3d31c5bf8173d6fdf8387e39c60 Nov 01 14:22:23.072146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:23.072119 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-phggr"] Nov 01 14:22:23.550409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:23.550371 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" event={"ID":"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9","Type":"ContainerStarted","Data":"fae71f20266a7700ca5bf301e22f16a067e0a3d31c5bf8173d6fdf8387e39c60"} Nov 01 14:22:23.910586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:23.910470 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:24.536106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.536072 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:24.734060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.734027 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq"] Nov 01 14:22:24.738442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.738425 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:24.790565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.790479 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq"] Nov 01 14:22:24.839296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.839264 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tg4ns\"" Nov 01 14:22:24.906901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.906866 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwmf\" (UniqueName: \"kubernetes.io/projected/674764aa-33af-462b-b221-9cd96d8569ce-kube-api-access-dqwmf\") pod \"cert-manager-cainjector-5c57bbd4fc-z98kq\" (UID: \"674764aa-33af-462b-b221-9cd96d8569ce\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:24.907059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:24.906977 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/674764aa-33af-462b-b221-9cd96d8569ce-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z98kq\" (UID: \"674764aa-33af-462b-b221-9cd96d8569ce\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:25.008268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:25.008189 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwmf\" (UniqueName: \"kubernetes.io/projected/674764aa-33af-462b-b221-9cd96d8569ce-kube-api-access-dqwmf\") pod \"cert-manager-cainjector-5c57bbd4fc-z98kq\" (UID: \"674764aa-33af-462b-b221-9cd96d8569ce\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:25.008656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:25.008291 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/674764aa-33af-462b-b221-9cd96d8569ce-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z98kq\" (UID: \"674764aa-33af-462b-b221-9cd96d8569ce\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:25.064921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:25.064850 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/674764aa-33af-462b-b221-9cd96d8569ce-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-z98kq\" (UID: \"674764aa-33af-462b-b221-9cd96d8569ce\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:25.065059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:25.065003 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwmf\" (UniqueName: \"kubernetes.io/projected/674764aa-33af-462b-b221-9cd96d8569ce-kube-api-access-dqwmf\") pod \"cert-manager-cainjector-5c57bbd4fc-z98kq\" (UID: \"674764aa-33af-462b-b221-9cd96d8569ce\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:25.349617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:25.349538 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" Nov 01 14:22:25.741050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:25.741020 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq"] Nov 01 14:22:26.135935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:26.135897 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:26.302757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:26.302727 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:26.566636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:26.566595 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" event={"ID":"674764aa-33af-462b-b221-9cd96d8569ce","Type":"ContainerStarted","Data":"d224dfa3b8249fc84563d5ef33c09b3ca6900368942d6133202a99b445978e7f"} Nov 01 14:22:27.490815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:27.490754 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:27.572185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:27.572143 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" event={"ID":"58a5c2fa-1637-4f72-9ad2-1b6a7f9107b9","Type":"ContainerStarted","Data":"d0256733316bd388e50e3b1cffd21cf927b11cfa7374a9e382c08779603943e0"} Nov 01 14:22:27.572367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:27.572204 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:27.573621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:27.573601 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" event={"ID":"674764aa-33af-462b-b221-9cd96d8569ce","Type":"ContainerStarted","Data":"d4b74c4b420b207ba8ac832dc44589816c71cc98eea1932cebc2e8f8db15c801"} Nov 01 14:22:27.690552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:27.690498 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" podStartSLOduration=1.52340943 podStartE2EDuration="5.690484068s" totalImagesPullingTime="4.167074638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:27.688953253 +0000 UTC m=+286.747105103" watchObservedRunningTime="2026-11-01 14:22:27.690484068 +0000 UTC m=+286.748635912" Nov 01 14:22:27.803175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:27.803122 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-z98kq" podStartSLOduration=2.12061632 podStartE2EDuration="3.803108779s" totalImagesPullingTime="1.682492459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:27.801771391 +0000 UTC m=+286.859923241" watchObservedRunningTime="2026-11-01 14:22:27.803108779 +0000 UTC m=+286.861260629" Nov 01 14:22:28.806589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.806553 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95"] Nov 01 14:22:28.810656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.810633 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:28.863364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.863330 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95"] Nov 01 14:22:28.912016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.911984 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 14:22:28.912016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.912002 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 14:22:28.945714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.945679 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bd90dc7-2050-456d-bdae-94c06c818d8d-tmp\") pod \"jobset-operator-674cd59dc5-rrd95\" (UID: \"0bd90dc7-2050-456d-bdae-94c06c818d8d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:28.945869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.945734 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726kh\" (UniqueName: \"kubernetes.io/projected/0bd90dc7-2050-456d-bdae-94c06c818d8d-kube-api-access-726kh\") pod \"jobset-operator-674cd59dc5-rrd95\" (UID: \"0bd90dc7-2050-456d-bdae-94c06c818d8d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:28.954850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.954823 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-gczzd\"" Nov 01 14:22:28.964901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:28.964872 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:29.046491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.046457 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bd90dc7-2050-456d-bdae-94c06c818d8d-tmp\") pod \"jobset-operator-674cd59dc5-rrd95\" (UID: \"0bd90dc7-2050-456d-bdae-94c06c818d8d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:29.046679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.046501 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726kh\" (UniqueName: \"kubernetes.io/projected/0bd90dc7-2050-456d-bdae-94c06c818d8d-kube-api-access-726kh\") pod \"jobset-operator-674cd59dc5-rrd95\" (UID: \"0bd90dc7-2050-456d-bdae-94c06c818d8d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:29.046866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.046846 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0bd90dc7-2050-456d-bdae-94c06c818d8d-tmp\") pod \"jobset-operator-674cd59dc5-rrd95\" (UID: \"0bd90dc7-2050-456d-bdae-94c06c818d8d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:29.104840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.104770 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-726kh\" (UniqueName: \"kubernetes.io/projected/0bd90dc7-2050-456d-bdae-94c06c818d8d-kube-api-access-726kh\") pod \"jobset-operator-674cd59dc5-rrd95\" (UID: \"0bd90dc7-2050-456d-bdae-94c06c818d8d\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:29.118897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.118847 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:29.122817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.122799 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" Nov 01 14:22:29.507276 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:22:29.507246 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bd90dc7_2050_456d_bdae_94c06c818d8d.slice/crio-6eaf05ef4a8d787247c38e5dec44ea40741f030227719097fecb6617552addff WatchSource:0}: Error finding container 6eaf05ef4a8d787247c38e5dec44ea40741f030227719097fecb6617552addff: Status 404 returned error can't find the container with id 6eaf05ef4a8d787247c38e5dec44ea40741f030227719097fecb6617552addff Nov 01 14:22:29.509934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.509904 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95"] Nov 01 14:22:29.585910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:29.585873 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" event={"ID":"0bd90dc7-2050-456d-bdae-94c06c818d8d","Type":"ContainerStarted","Data":"6eaf05ef4a8d787247c38e5dec44ea40741f030227719097fecb6617552addff"} Nov 01 14:22:30.538698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:30.538667 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-6gn9k" Nov 01 14:22:31.106602 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:22:31.106558 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:22:31.294264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:31.294224 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:31.553591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:31.553552 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:32.175476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:32.175441 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:32.602877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:32.602835 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" event={"ID":"0bd90dc7-2050-456d-bdae-94c06c818d8d","Type":"ContainerStarted","Data":"623d52a81842eecaecf4c10ee821af85a60c6e5f5d9b63042d421ae9af67da12"} Nov 01 14:22:32.720784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:32.720724 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rrd95" podStartSLOduration=1.73740976 podStartE2EDuration="4.720712422s" totalImagesPullingTime="2.983302662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:32.719786294 +0000 UTC m=+291.777938143" watchObservedRunningTime="2026-11-01 14:22:32.720712422 +0000 UTC m=+291.778864271" Nov 01 14:22:32.896469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:32.896435 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:33.071229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.071191 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:33.292079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.292051 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:33.333501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.333469 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x"] Nov 01 14:22:33.337083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.337066 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.381290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.381262 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gx8\" (UniqueName: \"kubernetes.io/projected/d95c16e8-7021-40dd-b862-49ac7798725f-kube-api-access-c8gx8\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.381466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.381302 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d95c16e8-7021-40dd-b862-49ac7798725f-tmp\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.381466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.381333 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95c16e8-7021-40dd-b862-49ac7798725f-tls-certs\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.388977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.388939 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x"] Nov 01 14:22:33.389430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.389402 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 14:22:33.437482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.437447 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 14:22:33.437482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.437453 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-656df\"" Nov 01 14:22:33.437482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.437483 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 14:22:33.437686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.437456 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 14:22:33.481991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.481962 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gx8\" (UniqueName: \"kubernetes.io/projected/d95c16e8-7021-40dd-b862-49ac7798725f-kube-api-access-c8gx8\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.482160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.482009 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d95c16e8-7021-40dd-b862-49ac7798725f-tmp\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.482160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.482041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95c16e8-7021-40dd-b862-49ac7798725f-tls-certs\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.484371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.484349 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d95c16e8-7021-40dd-b862-49ac7798725f-tmp\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.484567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.484549 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d95c16e8-7021-40dd-b862-49ac7798725f-tls-certs\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.501547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.501515 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:33.538191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.538159 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gx8\" (UniqueName: \"kubernetes.io/projected/d95c16e8-7021-40dd-b862-49ac7798725f-kube-api-access-c8gx8\") pod \"kube-auth-proxy-7fc9ccddb-pkr2x\" (UID: \"d95c16e8-7021-40dd-b862-49ac7798725f\") " pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.581854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.581782 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-phggr" Nov 01 14:22:33.649200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.649174 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" Nov 01 14:22:33.935319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:33.935244 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:34.014791 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:22:34.014762 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95c16e8_7021_40dd_b862_49ac7798725f.slice/crio-d7d37126670b9679106ff2809550af2302acd6fe865d684b5b64f101453d9706 WatchSource:0}: Error finding container d7d37126670b9679106ff2809550af2302acd6fe865d684b5b64f101453d9706: Status 404 returned error can't find the container with id d7d37126670b9679106ff2809550af2302acd6fe865d684b5b64f101453d9706 Nov 01 14:22:34.016847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:34.016826 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x"] Nov 01 14:22:34.216170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:34.216084 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:34.613576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:34.613541 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" event={"ID":"d95c16e8-7021-40dd-b862-49ac7798725f","Type":"ContainerStarted","Data":"d7d37126670b9679106ff2809550af2302acd6fe865d684b5b64f101453d9706"} Nov 01 14:22:34.950775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:34.950688 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:35.126279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:35.126238 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:36.996251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:36.996200 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:37.492880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:37.492816 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:37.632018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:37.631977 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" event={"ID":"d95c16e8-7021-40dd-b862-49ac7798725f","Type":"ContainerStarted","Data":"72cdfd81847c6be09bfcca2b5421d22bd5f36e37f6f4d7147f2a5c5071f7fceb"} Nov 01 14:22:37.655701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:37.655666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:37.747430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:37.747329 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fc9ccddb-pkr2x" podStartSLOduration=1.6727770789999998 podStartE2EDuration="4.747316687s" totalImagesPullingTime="3.074539608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:37.747292396 +0000 UTC m=+296.805444272" watchObservedRunningTime="2026-11-01 14:22:37.747316687 +0000 UTC m=+296.805468536" Nov 01 14:22:38.095473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:38.095424 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:38.253778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:38.253717 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:40.129126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.129091 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:40.183191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.183160 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:22:40.186677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.186662 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.252272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.252239 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7n7t\" (UniqueName: \"kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t\") pod \"limitador-limitador-74dfcfb894-blhjv\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.252458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.252301 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file\") pod \"limitador-limitador-74dfcfb894-blhjv\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.266692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.266661 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:22:40.301623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.301577 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:40.339984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.339956 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 14:22:40.348057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.348028 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jxs9f\"" Nov 01 14:22:40.353559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.353526 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7n7t\" (UniqueName: \"kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t\") pod \"limitador-limitador-74dfcfb894-blhjv\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.353706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.353661 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file\") pod \"limitador-limitador-74dfcfb894-blhjv\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.354322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.354300 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file\") pod \"limitador-limitador-74dfcfb894-blhjv\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.411000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.410929 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7n7t\" (UniqueName: \"kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t\") pod \"limitador-limitador-74dfcfb894-blhjv\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.421928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.421898 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:22:40.450720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.450675 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:40.498825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.498791 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:40.619712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.619675 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:40.866956 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:22:40.866920 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e8039b_b0b3_43f1_9c2c_bfcd840323f2.slice/crio-308ed2a86e585b0c0d73d1ae4ea8f9f2d398ac82e531bc3993c220308a95dab6 WatchSource:0}: Error finding container 308ed2a86e585b0c0d73d1ae4ea8f9f2d398ac82e531bc3993c220308a95dab6: Status 404 returned error can't find the container with id 308ed2a86e585b0c0d73d1ae4ea8f9f2d398ac82e531bc3993c220308a95dab6 Nov 01 14:22:40.868593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.868549 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:22:40.923080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:40.923048 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:41.106483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:41.106458 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:22:41.107202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:41.107174 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:22:41.553739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:41.553688 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:41.651291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:41.651256 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" event={"ID":"75e8039b-b0b3-43f1-9c2c-bfcd840323f2","Type":"ContainerStarted","Data":"308ed2a86e585b0c0d73d1ae4ea8f9f2d398ac82e531bc3993c220308a95dab6"} Nov 01 14:22:41.710918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:41.710885 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:43.496179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:43.496092 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:43.662730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:43.662695 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:43.666406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:43.666375 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" event={"ID":"75e8039b-b0b3-43f1-9c2c-bfcd840323f2","Type":"ContainerStarted","Data":"b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9"} Nov 01 14:22:43.666537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:43.666461 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:43.780596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:43.780538 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" podStartSLOduration=1.5703897169999999 podStartE2EDuration="3.780519736s" totalImagesPullingTime="2.210130019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:22:43.779078058 +0000 UTC m=+302.837229909" watchObservedRunningTime="2026-11-01 14:22:43.780519736 +0000 UTC m=+302.838671587" Nov 01 14:22:44.090094 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:44.090014 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:44.284557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:44.284528 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:44.693121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:44.693089 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:44.860846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:44.860814 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:46.737688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:46.737634 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:51.892407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:51.892373 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:52.051678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:52.051641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:54.672262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:54.672233 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:22:56.694092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:56.694059 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:56.854346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:56.854309 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:22:59.374577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:59.374543 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:22:59.375034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:59.374730 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" podUID="75e8039b-b0b3-43f1-9c2c-bfcd840323f2" containerName="limitador" containerID="cri-o://b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9" gracePeriod=30 Nov 01 14:22:59.911458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:22:59.911435 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:23:00.018411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.018382 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file\" (UniqueName: \"kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file\") pod \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " Nov 01 14:23:00.018596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.018434 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t\" (UniqueName: \"kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t\") pod \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\" (UID: \"75e8039b-b0b3-43f1-9c2c-bfcd840323f2\") " Nov 01 14:23:00.018799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.018775 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file" pod "75e8039b-b0b3-43f1-9c2c-bfcd840323f2" (UID: "75e8039b-b0b3-43f1-9c2c-bfcd840323f2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:23:00.020861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.020832 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t" pod "75e8039b-b0b3-43f1-9c2c-bfcd840323f2" (UID: "75e8039b-b0b3-43f1-9c2c-bfcd840323f2"). InnerVolumeSpecName "kube-api-access-z7n7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:23:00.119328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.119291 5473 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-config-file\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:23:00.119328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.119324 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7n7t\" (UniqueName: \"kubernetes.io/projected/75e8039b-b0b3-43f1-9c2c-bfcd840323f2-kube-api-access-z7n7t\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:23:00.494959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.494861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:00.652267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.652231 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:00.744773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.744739 5473 generic.go:356] "Generic (PLEG): container finished" podID="75e8039b-b0b3-43f1-9c2c-bfcd840323f2" containerID="b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9" exitCode=0 Nov 01 14:23:00.744952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.744789 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" Nov 01 14:23:00.744952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.744825 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" event={"ID":"75e8039b-b0b3-43f1-9c2c-bfcd840323f2","Type":"ContainerDied","Data":"b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9"} Nov 01 14:23:00.744952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.744878 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-blhjv" event={"ID":"75e8039b-b0b3-43f1-9c2c-bfcd840323f2","Type":"ContainerDied","Data":"308ed2a86e585b0c0d73d1ae4ea8f9f2d398ac82e531bc3993c220308a95dab6"} Nov 01 14:23:00.744952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.744900 5473 scope.go:122] "RemoveContainer" containerID="b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9" Nov 01 14:23:00.755051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.755022 5473 scope.go:122] "RemoveContainer" containerID="b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9" Nov 01 14:23:00.755351 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:23:00.755332 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9\": container with ID starting with b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9 not found: ID does not exist" containerID="b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9" Nov 01 14:23:00.755423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.755360 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9"} err="failed to get container status \"b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9\": rpc error: code = NotFound desc = could not find container \"b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9\": container with ID starting with b53f1326437d0484b6f578fdff6d7d4b98a39c27af59f2895c210f66418609a9 not found: ID does not exist" Nov 01 14:23:00.818090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.818056 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:00.908350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.908315 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:23:00.913841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:00.913813 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-blhjv"] Nov 01 14:23:01.080637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:01.080604 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e8039b-b0b3-43f1-9c2c-bfcd840323f2" path="/var/lib/kubelet/pods/75e8039b-b0b3-43f1-9c2c-bfcd840323f2/volumes" Nov 01 14:23:01.105951 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:23:01.105915 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:23:03.827515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:03.827482 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:04.045802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.045764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:04.215159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.215051 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:04.368382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.368351 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:04.412383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.412352 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29"] Nov 01 14:23:04.412725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.412712 5473 cpu_manager.go:414] "containerMap: removing container" podUID="75e8039b-b0b3-43f1-9c2c-bfcd840323f2" containerName="limitador" Nov 01 14:23:04.412772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.412727 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e8039b-b0b3-43f1-9c2c-bfcd840323f2" containerName="limitador" Nov 01 14:23:04.412806 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.412784 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e8039b-b0b3-43f1-9c2c-bfcd840323f2" containerName="limitador" Nov 01 14:23:04.418605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.418584 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.455591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.455562 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkj9\" (UniqueName: \"kubernetes.io/projected/a124ad3b-6c83-45d3-ac07-2ff5b7e690ba-kube-api-access-4rkj9\") pod \"postgres-7cdbf6dc44-5rq29\" (UID: \"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.455591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.455599 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a124ad3b-6c83-45d3-ac07-2ff5b7e690ba-data\") pod \"postgres-7cdbf6dc44-5rq29\" (UID: \"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.468320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.468255 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29"] Nov 01 14:23:04.470760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.470733 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 14:23:04.480232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.480200 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jq5zl\"" Nov 01 14:23:04.516207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.516180 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 14:23:04.516334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.516245 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 14:23:04.537286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.537253 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:04.556496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.556468 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkj9\" (UniqueName: \"kubernetes.io/projected/a124ad3b-6c83-45d3-ac07-2ff5b7e690ba-kube-api-access-4rkj9\") pod \"postgres-7cdbf6dc44-5rq29\" (UID: \"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.556611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.556502 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a124ad3b-6c83-45d3-ac07-2ff5b7e690ba-data\") pod \"postgres-7cdbf6dc44-5rq29\" (UID: \"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.556926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.556908 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a124ad3b-6c83-45d3-ac07-2ff5b7e690ba-data\") pod \"postgres-7cdbf6dc44-5rq29\" (UID: \"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.615074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.615039 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkj9\" (UniqueName: \"kubernetes.io/projected/a124ad3b-6c83-45d3-ac07-2ff5b7e690ba-kube-api-access-4rkj9\") pod \"postgres-7cdbf6dc44-5rq29\" (UID: \"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.700955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.700925 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:04.732596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.732529 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:04.858798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:04.858760 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:05.096886 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:23:05.096857 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda124ad3b_6c83_45d3_ac07_2ff5b7e690ba.slice/crio-22c0126ace484f082cef9702534e1ad7d8f372457cc92dd440467c4820e34baf WatchSource:0}: Error finding container 22c0126ace484f082cef9702534e1ad7d8f372457cc92dd440467c4820e34baf: Status 404 returned error can't find the container with id 22c0126ace484f082cef9702534e1ad7d8f372457cc92dd440467c4820e34baf Nov 01 14:23:05.098133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:05.098110 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29"] Nov 01 14:23:05.098557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:05.098537 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 14:23:05.687370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:05.687336 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:05.770704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:05.770674 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" event={"ID":"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba","Type":"ContainerStarted","Data":"22c0126ace484f082cef9702534e1ad7d8f372457cc92dd440467c4820e34baf"} Nov 01 14:23:05.855798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:05.855764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:06.671018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.670981 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j"] Nov 01 14:23:06.675920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.675892 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.726800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.726767 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j"] Nov 01 14:23:06.775301 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.775270 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 14:23:06.775482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.775411 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64e06279-a1c7-4003-9e4f-d22e967ca055-cert\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.775482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.775470 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/64e06279-a1c7-4003-9e4f-d22e967ca055-manager-config\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.775583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.775554 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e06279-a1c7-4003-9e4f-d22e967ca055-metrics-certs\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.775630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.775604 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bbq\" (UniqueName: \"kubernetes.io/projected/64e06279-a1c7-4003-9e4f-d22e967ca055-kube-api-access-d8bbq\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.778010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.777988 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-4g2c8\"" Nov 01 14:23:06.778010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.777999 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 14:23:06.778159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.778031 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 14:23:06.876819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.876785 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e06279-a1c7-4003-9e4f-d22e967ca055-metrics-certs\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.877007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.876830 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bbq\" (UniqueName: \"kubernetes.io/projected/64e06279-a1c7-4003-9e4f-d22e967ca055-kube-api-access-d8bbq\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.877007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.876956 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64e06279-a1c7-4003-9e4f-d22e967ca055-cert\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.877132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.877056 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/64e06279-a1c7-4003-9e4f-d22e967ca055-manager-config\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.877727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.877706 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/64e06279-a1c7-4003-9e4f-d22e967ca055-manager-config\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.879300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.879278 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64e06279-a1c7-4003-9e4f-d22e967ca055-metrics-certs\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.879430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.879411 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/64e06279-a1c7-4003-9e4f-d22e967ca055-cert\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.933684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.933619 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bbq\" (UniqueName: \"kubernetes.io/projected/64e06279-a1c7-4003-9e4f-d22e967ca055-kube-api-access-d8bbq\") pod \"jobset-controller-manager-666cf69fc7-rsc5j\" (UID: \"64e06279-a1c7-4003-9e4f-d22e967ca055\") " pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:06.988392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:06.988362 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:07.357281 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:23:07.357251 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e06279_a1c7_4003_9e4f_d22e967ca055.slice/crio-ac0b5dc9613acab78782c1ebb8b58617d9d00b9c9d47b3e6872cead067c1f8e4 WatchSource:0}: Error finding container ac0b5dc9613acab78782c1ebb8b58617d9d00b9c9d47b3e6872cead067c1f8e4: Status 404 returned error can't find the container with id ac0b5dc9613acab78782c1ebb8b58617d9d00b9c9d47b3e6872cead067c1f8e4 Nov 01 14:23:07.359129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:07.359099 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j"] Nov 01 14:23:07.784682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:07.784599 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" event={"ID":"64e06279-a1c7-4003-9e4f-d22e967ca055","Type":"ContainerStarted","Data":"ac0b5dc9613acab78782c1ebb8b58617d9d00b9c9d47b3e6872cead067c1f8e4"} Nov 01 14:23:11.811177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:11.811109 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" event={"ID":"64e06279-a1c7-4003-9e4f-d22e967ca055","Type":"ContainerStarted","Data":"ab4ebfba612c5ef5b12d592b0387e69c99d91d9a4e025a9e30da52dc92411d5f"} Nov 01 14:23:11.811619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:11.811231 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:11.925121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:11.925022 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" podStartSLOduration=1.632615618 podStartE2EDuration="5.925006562s" totalImagesPullingTime="4.292390944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:23:11.923738214 +0000 UTC m=+330.981890066" watchObservedRunningTime="2026-11-01 14:23:11.925006562 +0000 UTC m=+330.983158413" Nov 01 14:23:12.092820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:12.092775 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:12.249489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:12.249405 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:12.817695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:12.817659 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" event={"ID":"a124ad3b-6c83-45d3-ac07-2ff5b7e690ba","Type":"ContainerStarted","Data":"4ac33929083c016a0c1d77795788433805fc4b96cd86f9d4b820122b7bb02e51"} Nov 01 14:23:12.931689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:12.931636 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" podStartSLOduration=2.062468381 podStartE2EDuration="8.931621602s" totalImagesPullingTime="6.869153221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:23:12.929846429 +0000 UTC m=+331.987998281" watchObservedRunningTime="2026-11-01 14:23:12.931621602 +0000 UTC m=+331.989773446" Nov 01 14:23:13.822331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:13.822287 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:16.466855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:16.466819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:16.627780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:16.627728 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:19.856857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:19.856831 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-5rq29" Nov 01 14:23:21.865693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:21.865641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:22.018419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:22.018379 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:22.294912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:22.294871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:22.479125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:22.479088 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:22.823364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:22.823335 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-666cf69fc7-rsc5j" Nov 01 14:23:26.455536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:26.455486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:26.615324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:26.615295 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:26.772861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:26.772832 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:26.932491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:26.932449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:30.870819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:30.870763 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:31.106187 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:23:31.106154 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:23:31.122984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:31.122885 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:32.906437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:32.906404 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:33.076268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:33.076174 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:33.757875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:33.757820 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:33.994729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:33.994680 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:23:44.620358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:23:44.620325 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:01.106642 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:24:01.106555 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:24:02.146897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:02.146863 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:02.408004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:02.407917 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:04.310759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:04.310726 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:04.469258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:04.469208 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:04.617928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:04.617843 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:05.868559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:05.868524 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.075104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.075069 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.222905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.222821 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.376241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.376199 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.530443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.530416 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.688593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.688555 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.866082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.865990 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:06.957749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.957712 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-gwmh8"] Nov 01 14:24:06.961252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:06.961237 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.007080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.007052 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-metrics-certs\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.007232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.007104 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sh9l\" (UniqueName: \"kubernetes.io/projected/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-kube-api-access-4sh9l\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.007232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.007188 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-cert\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.015189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.015111 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-gwmh8"] Nov 01 14:24:07.062722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.062691 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 14:24:07.063739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.063721 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 14:24:07.063864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.063827 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 14:24:07.063956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.063944 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-ktlqm\"" Nov 01 14:24:07.063997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.063967 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 14:24:07.107881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.107853 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-cert\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.108048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.107919 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-metrics-certs\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.108048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.107957 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sh9l\" (UniqueName: \"kubernetes.io/projected/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-kube-api-access-4sh9l\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.110352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.110329 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-cert\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.110455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.110430 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-metrics-certs\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.165094 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.165023 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sh9l\" (UniqueName: \"kubernetes.io/projected/2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7-kube-api-access-4sh9l\") pod \"odh-model-controller-769875db56-gwmh8\" (UID: \"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7\") " pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.273856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.273827 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:07.635179 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:24:07.635143 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ef0fc4f_c327_4c76_98ae_b1d7bd97d8b7.slice/crio-3a9d3e96b0cf5b0e41b33be8d614dc596a706b4ea151dab1e20c224a1455a036 WatchSource:0}: Error finding container 3a9d3e96b0cf5b0e41b33be8d614dc596a706b4ea151dab1e20c224a1455a036: Status 404 returned error can't find the container with id 3a9d3e96b0cf5b0e41b33be8d614dc596a706b4ea151dab1e20c224a1455a036 Nov 01 14:24:07.637357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.637337 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-gwmh8"] Nov 01 14:24:07.920579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:07.920492 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:08.084417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:08.084388 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-gwmh8" event={"ID":"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7","Type":"ContainerStarted","Data":"3a9d3e96b0cf5b0e41b33be8d614dc596a706b4ea151dab1e20c224a1455a036"} Nov 01 14:24:11.100126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:11.100086 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-gwmh8" event={"ID":"2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7","Type":"ContainerStarted","Data":"4bb2c539790fdf89ffadc67a4c64d2499c336ab316613332a7bb5d8acb01a03e"} Nov 01 14:24:11.100595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:11.100247 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:11.213338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:11.213237 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-gwmh8" podStartSLOduration=1.900404899 podStartE2EDuration="5.213199837s" totalImagesPullingTime="3.312794938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:24:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:24:11.212286526 +0000 UTC m=+390.270438376" watchObservedRunningTime="2026-11-01 14:24:11.213199837 +0000 UTC m=+390.271351687" Nov 01 14:24:12.582894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:12.582858 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:12.785971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:12.785938 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:12.938999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:12.938896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:13.083829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:13.083801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:13.249929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:13.249829 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:13.408400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:13.408354 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:13.566110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:13.566075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:13.725872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:13.725806 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:15.353091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:15.353060 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:15.570130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:15.570079 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:22.108425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:22.108396 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-gwmh8" Nov 01 14:24:31.107022 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:24:31.106985 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:24:44.912875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:44.912837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:54.757608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:54.757575 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:24:54.947577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:24:54.947535 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:01.107006 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:25:01.106972 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:25:03.751847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:03.751812 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:03.908611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:03.908582 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:31.106571 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:25:31.106495 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:25:43.168463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:43.168421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:45.227105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:45.227070 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:45.385437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:45.385403 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:55.499477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:55.499438 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:25:55.663811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:25:55.663769 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:01.106682 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:26:01.106649 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:26:02.215819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:02.215785 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:02.443300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:02.443268 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:24.676078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:24.676042 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:25.890162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:25.890130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:26.239673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:26.239587 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:26.404759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:26.404725 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:31.106307 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:26:31.106269 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:26:32.512310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:32.512272 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:41.498794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.498762 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8"] Nov 01 14:26:41.501969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.501952 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.556062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.556025 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8"] Nov 01 14:26:41.603853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.603823 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 14:26:41.603998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.603881 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 14:26:41.603998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.603893 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cbdjr\"" Nov 01 14:26:41.604665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.604646 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 14:26:41.617561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.617535 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:41.621719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.621701 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.621824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.621727 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.621824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.621750 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.621903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.621856 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4c309d-c000-4d8e-a84b-e5e132765d60-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.621903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.621879 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprts\" (UniqueName: \"kubernetes.io/projected/fa4c309d-c000-4d8e-a84b-e5e132765d60-kube-api-access-wprts\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.621976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.621904 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722397 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722445 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722469 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722497 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722569 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4c309d-c000-4d8e-a84b-e5e132765d60-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722601 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprts\" (UniqueName: \"kubernetes.io/projected/fa4c309d-c000-4d8e-a84b-e5e132765d60-kube-api-access-wprts\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722866 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722903 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.722952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.722913 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.724818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.724799 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4c309d-c000-4d8e-a84b-e5e132765d60-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.725085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.725066 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4c309d-c000-4d8e-a84b-e5e132765d60-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.779133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.779105 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprts\" (UniqueName: \"kubernetes.io/projected/fa4c309d-c000-4d8e-a84b-e5e132765d60-kube-api-access-wprts\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k4fn8\" (UID: \"fa4c309d-c000-4d8e-a84b-e5e132765d60\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:41.814137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:41.814112 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:42.177352 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:26:42.177325 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4c309d_c000_4d8e_a84b_e5e132765d60.slice/crio-2980af18ee15c8aadd8a94172b7872d4437a60c42c1702573709771b2f2bf9c1 WatchSource:0}: Error finding container 2980af18ee15c8aadd8a94172b7872d4437a60c42c1702573709771b2f2bf9c1: Status 404 returned error can't find the container with id 2980af18ee15c8aadd8a94172b7872d4437a60c42c1702573709771b2f2bf9c1 Nov 01 14:26:42.179071 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:42.179048 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8"] Nov 01 14:26:42.807133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:42.807101 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" event={"ID":"fa4c309d-c000-4d8e-a84b-e5e132765d60","Type":"ContainerStarted","Data":"2980af18ee15c8aadd8a94172b7872d4437a60c42c1702573709771b2f2bf9c1"} Nov 01 14:26:43.814005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:43.813973 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" event={"ID":"fa4c309d-c000-4d8e-a84b-e5e132765d60","Type":"ContainerStarted","Data":"cb37fb065aeb869c1f2a63a3429b9e49081eacfa85139f0dd4632e41d4cf5bea"} Nov 01 14:26:43.814453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:43.814162 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:43.929609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:43.929559 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" podStartSLOduration=1.399855964 podStartE2EDuration="2.929544074s" totalImagesPullingTime="1.52968811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:43.927726999 +0000 UTC m=+542.985878849" watchObservedRunningTime="2026-11-01 14:26:43.929544074 +0000 UTC m=+542.987695924" Nov 01 14:26:44.356505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:44.356449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:45.536898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:45.536860 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:46.933487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:46.933455 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7"] Nov 01 14:26:46.937087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:46.937070 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:46.989898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:46.989863 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7"] Nov 01 14:26:47.037877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.037845 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 01 14:26:47.047228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.047186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:47.069443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.069410 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxp2\" (UniqueName: \"kubernetes.io/projected/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-kube-api-access-bzxp2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.069581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.069454 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.069581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.069481 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.069693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.069643 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.069750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.069717 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.069796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.069769 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.170595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.170556 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxp2\" (UniqueName: \"kubernetes.io/projected/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-kube-api-access-bzxp2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.170595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.170599 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.170848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.170630 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.170848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.170672 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.170848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.170714 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.170848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.170743 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.171136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.171113 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.171240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.171193 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.171289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.171271 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.172981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.172957 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.173153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.173135 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.229625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.229554 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxp2\" (UniqueName: \"kubernetes.io/projected/f2603428-d7ca-4679-9cf4-8cfdb207c5e9-kube-api-access-bzxp2\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-rw8z7\" (UID: \"f2603428-d7ca-4679-9cf4-8cfdb207c5e9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.249951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.249926 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:47.619071 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:26:47.619039 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2603428_d7ca_4679_9cf4_8cfdb207c5e9.slice/crio-88cd2aed705491f70361ddc045132cf97dcec56252e66d2b560bae6bb323d50e WatchSource:0}: Error finding container 88cd2aed705491f70361ddc045132cf97dcec56252e66d2b560bae6bb323d50e: Status 404 returned error can't find the container with id 88cd2aed705491f70361ddc045132cf97dcec56252e66d2b560bae6bb323d50e Nov 01 14:26:47.620669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.620645 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7"] Nov 01 14:26:47.833817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:47.833783 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" event={"ID":"f2603428-d7ca-4679-9cf4-8cfdb207c5e9","Type":"ContainerStarted","Data":"88cd2aed705491f70361ddc045132cf97dcec56252e66d2b560bae6bb323d50e"} Nov 01 14:26:48.839680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:48.839649 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" event={"ID":"f2603428-d7ca-4679-9cf4-8cfdb207c5e9","Type":"ContainerStarted","Data":"84e49bf43782998edb0318b0461d3bb8386198067aa5cc6eb45ab9fee7f65428"} Nov 01 14:26:48.954870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:48.954816 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" podStartSLOduration=2.35661313 podStartE2EDuration="2.954800537s" totalImagesPullingTime="598.187407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:48.95340996 +0000 UTC m=+548.011561810" watchObservedRunningTime="2026-11-01 14:26:48.954800537 +0000 UTC m=+548.012952387" Nov 01 14:26:49.250447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:49.250350 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:51.150051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:51.149967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:51.815040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:51.814997 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:51.827976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:51.827938 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:51.865722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:51.865689 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k4fn8" Nov 01 14:26:55.245814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.245781 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qcr5w"] Nov 01 14:26:55.249057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.249041 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.303004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.302972 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qcr5w"] Nov 01 14:26:55.341360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.341327 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.341538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.341377 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-home\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.341538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.341446 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.341625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.341558 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.341625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.341580 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6067073-4021-4e03-b1d7-03260b1618c9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.341625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.341612 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtl2v\" (UniqueName: \"kubernetes.io/projected/c6067073-4021-4e03-b1d7-03260b1618c9-kube-api-access-rtl2v\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.351168 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.351140 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 01 14:26:55.352457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.352426 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:55.442917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.442878 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-home\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.442917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.442914 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.442976 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.442992 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6067073-4021-4e03-b1d7-03260b1618c9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.443008 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtl2v\" (UniqueName: \"kubernetes.io/projected/c6067073-4021-4e03-b1d7-03260b1618c9-kube-api-access-rtl2v\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.443026 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.443388 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-home\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.443408 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.443486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.443471 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.445385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.445362 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6067073-4021-4e03-b1d7-03260b1618c9-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.445541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.445522 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6067073-4021-4e03-b1d7-03260b1618c9-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.501702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.501619 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtl2v\" (UniqueName: \"kubernetes.io/projected/c6067073-4021-4e03-b1d7-03260b1618c9-kube-api-access-rtl2v\") pod \"echo-simulator-kserve-64b6f8f96c-qcr5w\" (UID: \"c6067073-4021-4e03-b1d7-03260b1618c9\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.561591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.561551 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:26:55.928578 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:26:55.928545 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6067073_4021_4e03_b1d7_03260b1618c9.slice/crio-77f42ad751950c762e86a41bb840a38fa149750ccbfcbf5bc541c381a205bbcb WatchSource:0}: Error finding container 77f42ad751950c762e86a41bb840a38fa149750ccbfcbf5bc541c381a205bbcb: Status 404 returned error can't find the container with id 77f42ad751950c762e86a41bb840a38fa149750ccbfcbf5bc541c381a205bbcb Nov 01 14:26:55.929343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:55.929321 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qcr5w"] Nov 01 14:26:56.881295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:56.881255 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" event={"ID":"c6067073-4021-4e03-b1d7-03260b1618c9","Type":"ContainerStarted","Data":"77f42ad751950c762e86a41bb840a38fa149750ccbfcbf5bc541c381a205bbcb"} Nov 01 14:26:57.250854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:57.250772 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:57.266653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:57.266611 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:57.906793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:57.906761 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-rw8z7" Nov 01 14:26:58.599870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:58.599789 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:26:58.894627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:58.894534 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" event={"ID":"c6067073-4021-4e03-b1d7-03260b1618c9","Type":"ContainerStarted","Data":"bf62b88e49561eab2cf940398a6a752598a6f2a33b852fcc8d471f703c0d8cd7"} Nov 01 14:26:59.009542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:59.009494 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" podStartSLOduration=1.634527085 podStartE2EDuration="4.009479042s" totalImagesPullingTime="2.374951957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:26:59.008102617 +0000 UTC m=+558.066254466" watchObservedRunningTime="2026-11-01 14:26:59.009479042 +0000 UTC m=+558.067631084" Nov 01 14:26:59.562347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:26:59.562314 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:27:01.105612 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:27:01.105581 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:27:01.835005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.834972 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz"] Nov 01 14:27:01.838490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.838474 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:01.890905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.890874 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz"] Nov 01 14:27:01.939079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.939053 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 14:27:01.955922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.955887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:01.998723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.998690 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrwg\" (UniqueName: \"kubernetes.io/projected/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-kube-api-access-lzrwg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:01.998893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.998756 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:01.998893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.998861 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:01.999007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.998911 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:01.999007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.998933 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:01.999007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:01.998987 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.099585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.099501 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.099585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.099564 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.099802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.099608 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.099802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.099633 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.099802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.099654 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.099802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.099695 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrwg\" (UniqueName: \"kubernetes.io/projected/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-kube-api-access-lzrwg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.100077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.100053 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.100122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.100086 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.100122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.100099 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.101827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.101806 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.101992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.101974 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.156736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.156702 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrwg\" (UniqueName: \"kubernetes.io/projected/8dcdd7f9-e49a-4a88-a4af-d888fc97551d-kube-api-access-lzrwg\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz\" (UID: \"8dcdd7f9-e49a-4a88-a4af-d888fc97551d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.450169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.450092 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:02.811481 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:27:02.811451 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dcdd7f9_e49a_4a88_a4af_d888fc97551d.slice/crio-22bf85750a03160ab8bacf0db8960fdf363c544d0b1ae4c6456dd563cdac2462 WatchSource:0}: Error finding container 22bf85750a03160ab8bacf0db8960fdf363c544d0b1ae4c6456dd563cdac2462: Status 404 returned error can't find the container with id 22bf85750a03160ab8bacf0db8960fdf363c544d0b1ae4c6456dd563cdac2462 Nov 01 14:27:02.813484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.813456 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz"] Nov 01 14:27:02.920320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:02.920284 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" event={"ID":"8dcdd7f9-e49a-4a88-a4af-d888fc97551d","Type":"ContainerStarted","Data":"22bf85750a03160ab8bacf0db8960fdf363c544d0b1ae4c6456dd563cdac2462"} Nov 01 14:27:03.926484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:03.926447 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" event={"ID":"8dcdd7f9-e49a-4a88-a4af-d888fc97551d","Type":"ContainerStarted","Data":"a19defc3637d647a66c4ccc4500f52877811640ba8b8ee83dab9c219cc8790aa"} Nov 01 14:27:04.039954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:04.039891 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podStartSLOduration=2.420873514 podStartE2EDuration="3.039878199s" totalImagesPullingTime="619.004685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:04.03884983 +0000 UTC m=+563.097001680" watchObservedRunningTime="2026-11-01 14:27:04.039878199 +0000 UTC m=+563.098030043" Nov 01 14:27:04.450535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:04.450501 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:05.562308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:05.562269 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:27:05.574990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:05.574962 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:27:05.946875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:05.946806 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qcr5w" Nov 01 14:27:12.450484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:12.450440 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:12.463039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:12.463013 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:12.987538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:12.987507 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:27:13.747669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:13.747636 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:13.909887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:13.909856 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:14.906267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.906236 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk"] Nov 01 14:27:14.909947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.909931 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919092 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919157 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919259 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919354 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919381 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919416 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzz6\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919464 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.919563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.919494 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:14.962415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:14.962384 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58vzm\"" Nov 01 14:27:15.012586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.012560 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 14:27:15.014601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.014583 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk"] Nov 01 14:27:15.020380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020356 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020432 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020470 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020493 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020620 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020656 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020757 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzz6\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020801 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020838 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.020897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.020861 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.021059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.021040 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.021263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.021240 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.021460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.021309 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.021547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.021454 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.023067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.023040 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.023203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.023186 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.077382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.077340 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.077382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.077367 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzz6\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pl9mk\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.222280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.222190 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:15.633722 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:27:15.633693 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe9ea6d_c5d1_41e2_9620_639f1e19b2f2.slice/crio-25a227f0951e9a14ebe56d60640dfa4382b54a3083a126a78485830ecc11aaca WatchSource:0}: Error finding container 25a227f0951e9a14ebe56d60640dfa4382b54a3083a126a78485830ecc11aaca: Status 404 returned error can't find the container with id 25a227f0951e9a14ebe56d60640dfa4382b54a3083a126a78485830ecc11aaca Nov 01 14:27:15.635503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.635396 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk"] Nov 01 14:27:15.987817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:15.987722 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" event={"ID":"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2","Type":"ContainerStarted","Data":"25a227f0951e9a14ebe56d60640dfa4382b54a3083a126a78485830ecc11aaca"} Nov 01 14:27:16.650141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:16.650111 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:19.753878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:19.753848 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:19.754182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:19.753914 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:19.754182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:19.753943 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:20.011671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.011585 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" event={"ID":"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2","Type":"ContainerStarted","Data":"738b4ccaa5387b5b42792147808ddc868e785df637ca7396e56047d406947356"} Nov 01 14:27:20.134633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.134572 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podStartSLOduration=2.017560286 podStartE2EDuration="6.134558445s" totalImagesPullingTime="4.116998159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:20.132723366 +0000 UTC m=+579.190875217" watchObservedRunningTime="2026-11-01 14:27:20.134558445 +0000 UTC m=+579.192710296" Nov 01 14:27:20.223242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.223184 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:20.224825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.224768 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:20.225134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.225082 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:20.685447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.685413 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:20.866903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:20.866869 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:21.222856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:21.222821 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:21.223031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:21.222876 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:22.223290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:22.223253 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:22.223771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:22.223311 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:23.222959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:23.222919 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:23.223164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:23.222984 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:24.223505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:24.223470 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:24.223927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:24.223527 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:25.222585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:25.222543 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:25.222764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:25.222603 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:26.223416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:26.223378 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 01 14:27:26.223934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:26.223427 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 01 14:27:27.281914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:27.281870 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:27.977287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:27.977257 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:27.977596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:27.977566 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:27.978446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:27.978426 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:27:29.087451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:29.087411 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:29.329807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:29.329772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:30.011068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.011034 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5"] Nov 01 14:27:30.019464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.019447 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062653 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062708 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062742 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062823 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062854 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062888 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062940 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swjjg\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062970 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.069238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.062993 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.127683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.127646 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5"] Nov 01 14:27:30.163803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.163769 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.163982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.163815 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.163982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.163854 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swjjg\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164086 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.163991 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164086 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164044 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164092 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164118 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164155 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164277 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164362 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164417 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164507 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164604 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.164780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.164763 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.166264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.166239 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.166345 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.166304 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.220367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.220342 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.220628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.220610 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swjjg\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg\") pod \"maas-default-gateway-openshift-default-7f5667f86c-nhjm5\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.332748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.332718 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:30.535243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.535195 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:30.696864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.696727 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:30.794904 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:27:30.794829 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de66753_3f67_4f73_b5dd_514858690bc7.slice/crio-46834cdb2d5f8c5ff8bcec524e48031cb17da08a44993c244691b8b77ed38c6b WatchSource:0}: Error finding container 46834cdb2d5f8c5ff8bcec524e48031cb17da08a44993c244691b8b77ed38c6b: Status 404 returned error can't find the container with id 46834cdb2d5f8c5ff8bcec524e48031cb17da08a44993c244691b8b77ed38c6b Nov 01 14:27:30.797390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.797366 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5"] Nov 01 14:27:30.798417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.798384 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:30.798538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.798463 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:30.798538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:30.798510 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:31.107331 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:27:31.107299 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:27:31.107646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:31.107623 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" event={"ID":"4de66753-3f67-4f73-b5dd-514858690bc7","Type":"ContainerStarted","Data":"ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c"} Nov 01 14:27:31.107727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:31.107672 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" event={"ID":"4de66753-3f67-4f73-b5dd-514858690bc7","Type":"ContainerStarted","Data":"46834cdb2d5f8c5ff8bcec524e48031cb17da08a44993c244691b8b77ed38c6b"} Nov 01 14:27:31.230316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:31.230263 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podStartSLOduration=2.230248411 podStartE2EDuration="2.230248411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:27:31.228499544 +0000 UTC m=+590.286651397" watchObservedRunningTime="2026-11-01 14:27:31.230248411 +0000 UTC m=+590.288400260" Nov 01 14:27:31.335240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:31.333136 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:31.349244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:31.340690 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:31.349244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:31.340826 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:32.344239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:32.343584 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:32.344239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:32.343641 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:33.344239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:33.339597 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:33.344239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:33.339662 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:34.346357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:34.346319 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:34.346933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:34.346901 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:35.340651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:35.340616 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:35.340946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:35.340919 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:36.349258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:36.344601 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:36.349258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:36.344665 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:37.333274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:37.333237 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-nhjm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 01 14:27:37.333454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:37.333301 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 01 14:27:39.038485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:39.038450 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:39.039328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:39.039305 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:39.039913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:39.039897 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:27:41.145265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:41.145236 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:27:41.146169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:41.146151 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:27:45.916115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:45.916052 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:53.590459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:53.590420 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:53.755181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:53.755147 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:58.040881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.040842 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:58.231235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.231160 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:58.550677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.550646 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:58.671719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.671684 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd"] Nov 01 14:27:58.675304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.675288 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.731643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.731614 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e27521d-openshift-default-dockercfg-dtk9p\"" Nov 01 14:27:58.783432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.783403 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd"] Nov 01 14:27:58.801833 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.801748 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:58.822034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822006 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822047 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822068 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822235 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822272 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzq9r\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822352 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822392 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.822444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.822419 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923274 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923311 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923349 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923422 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923510 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923550 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923592 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzq9r\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923624 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923835 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.923910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.923885 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.924183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.924156 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.924183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.924169 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.924411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.924283 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.925775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.925753 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.925965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.925948 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.978659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.978624 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:58.981529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.981487 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzq9r\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.981698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.981634 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token\") pod \"e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:58.987752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:58.987726 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:27:59.137144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.137112 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:59.250903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.250839 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd"] Nov 01 14:27:59.254745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.254725 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.308641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.308557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:59.309714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.309685 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd"] Nov 01 14:27:59.310509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.310488 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c9fe7b2a-gw-openshift-default-dockercfg-rdmd8\"" Nov 01 14:27:59.409322 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:27:59.409295 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0533b77_9087_4fde_a8cf_f3481223bbd9.slice/crio-74e422591635177e527b4bdb92641899f42c86ae4998348f57a3d371dab29757 WatchSource:0}: Error finding container 74e422591635177e527b4bdb92641899f42c86ae4998348f57a3d371dab29757: Status 404 returned error can't find the container with id 74e422591635177e527b4bdb92641899f42c86ae4998348f57a3d371dab29757 Nov 01 14:27:59.411533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.411487 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd"] Nov 01 14:27:59.412119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.412086 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:59.412302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.412148 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:59.412302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.412176 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:27:59.428782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.428759 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.428897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.428819 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.428962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.428930 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.429019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.428998 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.429077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.429034 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.429137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.429095 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.429200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.429137 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.429200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.429164 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.429200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.429192 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.483586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.483549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:59.530312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530279 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530328 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530361 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530400 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530440 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530463 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530498 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530535 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530604 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530755 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.530966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530828 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.531026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.530964 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.531084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.531059 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.531366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.531345 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.533065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.533036 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.533264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.533201 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.587262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.587184 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.587395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.587321 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np\") pod \"e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:27:59.633839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.633807 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:59.803470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.803435 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:27:59.867822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:27:59.867748 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:00.281910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.280910 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" event={"ID":"b0533b77-9087-4fde-a8cf-f3481223bbd9","Type":"ContainerStarted","Data":"1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8"} Nov 01 14:28:00.281910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.280974 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" event={"ID":"b0533b77-9087-4fde-a8cf-f3481223bbd9","Type":"ContainerStarted","Data":"74e422591635177e527b4bdb92641899f42c86ae4998348f57a3d371dab29757"} Nov 01 14:28:00.281910 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:00.281864 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8eb5a7_15aa_44f1_8738_a34081db3f80.slice/crio-ef00acf5c8e3610a705aa8f4d7a3cc3ae700036770b42601e04a1d8107472250 WatchSource:0}: Error finding container ef00acf5c8e3610a705aa8f4d7a3cc3ae700036770b42601e04a1d8107472250: Status 404 returned error can't find the container with id ef00acf5c8e3610a705aa8f4d7a3cc3ae700036770b42601e04a1d8107472250 Nov 01 14:28:00.283400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.283363 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:00.283537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.283425 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd"] Nov 01 14:28:00.283537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.283439 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:00.283537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.283486 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:00.988660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.988631 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:28:00.993081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:00.993057 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:28:01.040267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.040233 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:01.107349 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:28:01.107314 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:28:01.112959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.112907 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podStartSLOduration=3.112894467 podStartE2EDuration="3.112894467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:00.402861462 +0000 UTC m=+619.461013311" watchObservedRunningTime="2026-11-01 14:28:01.112894467 +0000 UTC m=+620.171046316" Nov 01 14:28:01.240848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.240756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:01.288450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.288412 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" event={"ID":"0f8eb5a7-15aa-44f1-8738-a34081db3f80","Type":"ContainerStarted","Data":"af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595"} Nov 01 14:28:01.288890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.288505 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" event={"ID":"0f8eb5a7-15aa-44f1-8738-a34081db3f80","Type":"ContainerStarted","Data":"ef00acf5c8e3610a705aa8f4d7a3cc3ae700036770b42601e04a1d8107472250"} Nov 01 14:28:01.288890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.288653 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:28:01.289524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.289505 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:28:01.403612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.403561 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podStartSLOduration=2.403545383 podStartE2EDuration="2.403545383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:01.402850498 +0000 UTC m=+620.461002347" watchObservedRunningTime="2026-11-01 14:28:01.403545383 +0000 UTC m=+620.461697234" Nov 01 14:28:01.868887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.868858 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:01.873586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:01.873561 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:02.294184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:02.294157 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:02.294977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:02.294961 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:02.396871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:02.396845 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:02.565547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:02.565294 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:02.717008 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:02.716978 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:03.161511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:03.161475 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:03.377648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:03.377618 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:03.565829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:03.565797 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:04.766706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:04.766670 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:04.933342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:04.933305 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:05.096685 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.096589 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:05.485054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.484981 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:05.508855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.508823 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv"] Nov 01 14:28:05.513514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.513497 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587411 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587446 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587579 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trk7g\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587622 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587649 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587685 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587767 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587811 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.587959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.587836 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.591277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.591257 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c0f3259b-openshift-default-dockercfg-fzntm\"" Nov 01 14:28:05.619275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.619248 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv"] Nov 01 14:28:05.650502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.650470 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:05.688667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688637 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trk7g\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.688824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688685 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.688824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688714 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.688824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688736 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.688824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688762 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.688824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688791 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.688824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688807 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688846 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.688878 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.689104 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.689239 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.689321 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.689359 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.689835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.689811 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.690992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.690973 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.691226 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.691200 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.745794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.745722 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trk7g\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.745794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.745723 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token\") pod \"e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.826328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.826294 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:05.851891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:05.851854 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:06.254796 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:06.254768 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de45ce9_12db_4799_9d61_525388e0a069.slice/crio-d8acde460c1f8e1a02d6113233879ddbcbd79c98a0d6f8b1d28deb9ecee0228d WatchSource:0}: Error finding container d8acde460c1f8e1a02d6113233879ddbcbd79c98a0d6f8b1d28deb9ecee0228d: Status 404 returned error can't find the container with id d8acde460c1f8e1a02d6113233879ddbcbd79c98a0d6f8b1d28deb9ecee0228d Nov 01 14:28:06.257056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.257023 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:06.257157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.257089 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:06.257157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.257123 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:06.257601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.257518 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv"] Nov 01 14:28:06.319612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.319583 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" event={"ID":"7de45ce9-12db-4799-9d61-525388e0a069","Type":"ContainerStarted","Data":"d8acde460c1f8e1a02d6113233879ddbcbd79c98a0d6f8b1d28deb9ecee0228d"} Nov 01 14:28:06.781419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.781388 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:06.950887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:06.950855 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:07.120282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.120178 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:07.326530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.326494 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" event={"ID":"7de45ce9-12db-4799-9d61-525388e0a069","Type":"ContainerStarted","Data":"e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2"} Nov 01 14:28:07.400376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.400285 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:07.485857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.485804 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podStartSLOduration=2.4857902689999998 podStartE2EDuration="2.485790269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:07.483712103 +0000 UTC m=+626.541863953" watchObservedRunningTime="2026-11-01 14:28:07.485790269 +0000 UTC m=+626.543942119" Nov 01 14:28:07.559340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.559306 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:07.726242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.726146 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:07.827267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.827207 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:07.832368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.832346 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:07.895493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:07.895460 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:08.331974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:08.331940 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:08.332952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:08.332928 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:08.491131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:08.491100 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:08.641528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:08.641449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:08.803832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:08.803799 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:09.982926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:09.982898 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:10.146294 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:10.146261 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:12.271314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.271279 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:12.437112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.437086 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:12.655815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.655736 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm"] Nov 01 14:28:12.658538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.658507 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:12.659439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.659425 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.717489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.717041 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm"] Nov 01 14:28:12.755672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.755639 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx26b\" (UniqueName: \"kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b\") pod \"payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.755672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.755678 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.762851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.762823 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d8f1c66f\"" Nov 01 14:28:12.762986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.762827 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d8f1c66f-dockercfg-6dxfb\"" Nov 01 14:28:12.822586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.822558 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j"] Nov 01 14:28:12.826189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.826164 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:12.856424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.856393 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx26b\" (UniqueName: \"kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b\") pod \"payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.856583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.856430 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.856583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.856508 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume\") pod \"payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:12.856583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.856547 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftrz\" (UniqueName: \"kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz\") pod \"payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:12.857382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.857360 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.866090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.866065 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:12.885917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.885888 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j"] Nov 01 14:28:12.920106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.920045 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx26b\" (UniqueName: \"kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b\") pod \"payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:12.957092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.957060 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftrz\" (UniqueName: \"kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz\") pod \"payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:12.957273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.957255 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume\") pod \"payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:12.957860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.957841 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume\") pod \"payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:12.972282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:12.972260 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:13.018168 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.018140 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftrz\" (UniqueName: \"kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz\") pod \"payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:13.079586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.079556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:13.140806 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.140777 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:13.343103 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:13.343068 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ddd089_3c38_4334_bce3_6722f2d7a216.slice/crio-bb2c1d05d7b759e14f0660b9999c5a4f1e4dd35f952ee11205513caeb13c6f74 WatchSource:0}: Error finding container bb2c1d05d7b759e14f0660b9999c5a4f1e4dd35f952ee11205513caeb13c6f74: Status 404 returned error can't find the container with id bb2c1d05d7b759e14f0660b9999c5a4f1e4dd35f952ee11205513caeb13c6f74 Nov 01 14:28:13.345458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.345436 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm"] Nov 01 14:28:13.345553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.345537 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 14:28:13.360997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.360972 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" event={"ID":"e6ddd089-3c38-4334-bce3-6722f2d7a216","Type":"ContainerStarted","Data":"bb2c1d05d7b759e14f0660b9999c5a4f1e4dd35f952ee11205513caeb13c6f74"} Nov 01 14:28:13.406244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.406173 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:13.505656 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:13.505621 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c4fb1b_ff2e_4ab6_a704_0203bc438439.slice/crio-f893cd6ed4cd171bd8986b7fb35923296c5fccefafb8020d28e921506c086757 WatchSource:0}: Error finding container f893cd6ed4cd171bd8986b7fb35923296c5fccefafb8020d28e921506c086757: Status 404 returned error can't find the container with id f893cd6ed4cd171bd8986b7fb35923296c5fccefafb8020d28e921506c086757 Nov 01 14:28:13.507662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.507639 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j"] Nov 01 14:28:13.577807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.577776 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:13.737515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:13.737476 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:14.367485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:14.367448 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" event={"ID":"48c4fb1b-ff2e-4ab6-a704-0203bc438439","Type":"ContainerStarted","Data":"f893cd6ed4cd171bd8986b7fb35923296c5fccefafb8020d28e921506c086757"} Nov 01 14:28:14.641630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:14.641545 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:14.807739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:14.807703 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:15.028040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.027997 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq"] Nov 01 14:28:15.032054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.032025 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.096105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.096061 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq"] Nov 01 14:28:15.099618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.099519 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98995\" (UniqueName: \"kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995\") pod \"payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.099618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.099571 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.142759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.142729 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c9fe7b2a\"" Nov 01 14:28:15.142759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.142735 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c9fe7b2a-dockercfg-54jk9\"" Nov 01 14:28:15.201114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.201079 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98995\" (UniqueName: \"kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995\") pod \"payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.201443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.201150 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.202608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.202455 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.258364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.258331 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98995\" (UniqueName: \"kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995\") pod \"payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.349653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.349569 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:15.734935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:15.734815 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq"] Nov 01 14:28:16.384600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:16.384559 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" event={"ID":"70e68d61-32a5-4ffe-987f-8a82e3181db0","Type":"ContainerStarted","Data":"971db1bc17a4da17c4bcc09fc7a55f4ea3005eefcb6337e0955e29dcd4533f75"} Nov 01 14:28:17.396633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.396597 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" event={"ID":"e6ddd089-3c38-4334-bce3-6722f2d7a216","Type":"ContainerStarted","Data":"6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc"} Nov 01 14:28:17.397165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.396735 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:17.398543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.398515 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" event={"ID":"48c4fb1b-ff2e-4ab6-a704-0203bc438439","Type":"ContainerStarted","Data":"8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a"} Nov 01 14:28:17.398675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.398567 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:17.400129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.400106 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" event={"ID":"70e68d61-32a5-4ffe-987f-8a82e3181db0","Type":"ContainerStarted","Data":"cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d"} Nov 01 14:28:17.400265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.400148 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:17.511362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.511314 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" podStartSLOduration=2.273086512 podStartE2EDuration="5.511298761s" totalImagesPullingTime="3.238212249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:17.510286312 +0000 UTC m=+636.568438163" watchObservedRunningTime="2026-11-01 14:28:17.511298761 +0000 UTC m=+636.569450629" Nov 01 14:28:17.743287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.743149 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" podStartSLOduration=2.558823607 podStartE2EDuration="5.743133098s" totalImagesPullingTime="3.184309491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:17.629914058 +0000 UTC m=+636.688065909" watchObservedRunningTime="2026-11-01 14:28:17.743133098 +0000 UTC m=+636.801284948" Nov 01 14:28:17.743651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:17.743618 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" podStartSLOduration=2.811402632 podStartE2EDuration="3.743609493s" totalImagesPullingTime="932.206861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:17.742448755 +0000 UTC m=+636.800600605" watchObservedRunningTime="2026-11-01 14:28:17.743609493 +0000 UTC m=+636.801761347" Nov 01 14:28:18.259015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:18.258978 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:18.418154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:18.418108 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:18.584770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:18.584739 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:23.078677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.078564 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:23.280941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.280909 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:23.446649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.446573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:23.610131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.610100 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:23.773645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.773609 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:23.893577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.893546 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw"] Nov 01 14:28:23.897099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.897082 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:23.953934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.953899 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw"] Nov 01 14:28:23.985703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.985670 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgh5v\" (UniqueName: \"kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v\") pod \"payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:23.985881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.985733 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:23.999524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.999494 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-c0f3259b\"" Nov 01 14:28:23.999644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:23.999584 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-c0f3259b-dockercfg-fgwvt\"" Nov 01 14:28:24.052365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.052298 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr"] Nov 01 14:28:24.056089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.056072 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.086766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.086736 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chq46\" (UniqueName: \"kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46\") pod \"payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.087209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.086802 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgh5v\" (UniqueName: \"kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v\") pod \"payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:24.087209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.086854 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:24.087209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.086912 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume\") pod \"payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.087765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.087742 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:24.109520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.109485 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr"] Nov 01 14:28:24.143499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.143476 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgh5v\" (UniqueName: \"kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v\") pod \"payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:24.187646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.187609 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chq46\" (UniqueName: \"kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46\") pod \"payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.187811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.187716 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume\") pod \"payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.188466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.188448 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume\") pod \"payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.210504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.210483 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:24.244026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.244001 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chq46\" (UniqueName: \"kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46\") pod \"payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.369874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.369785 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:24.576641 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:24.576612 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef693117_223e_41d2_b4ae_15b153d2580c.slice/crio-c5a6fcb6997a164ca509b42fc5f66e31adbc358799c6aa3cfd877f71b90b51f3 WatchSource:0}: Error finding container c5a6fcb6997a164ca509b42fc5f66e31adbc358799c6aa3cfd877f71b90b51f3: Status 404 returned error can't find the container with id c5a6fcb6997a164ca509b42fc5f66e31adbc358799c6aa3cfd877f71b90b51f3 Nov 01 14:28:24.578982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.578958 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw"] Nov 01 14:28:24.736662 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:24.736633 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44f04d0_36f2_48f1_a806_16bad817a5da.slice/crio-85cfeb00c6b92a279b979e1c32baa1ac2e754e0c728fe6fcbf6ca7a61b5a8588 WatchSource:0}: Error finding container 85cfeb00c6b92a279b979e1c32baa1ac2e754e0c728fe6fcbf6ca7a61b5a8588: Status 404 returned error can't find the container with id 85cfeb00c6b92a279b979e1c32baa1ac2e754e0c728fe6fcbf6ca7a61b5a8588 Nov 01 14:28:24.739703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:24.739679 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr"] Nov 01 14:28:25.451749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.451710 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" event={"ID":"d44f04d0-36f2-48f1-a806-16bad817a5da","Type":"ContainerStarted","Data":"65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84"} Nov 01 14:28:25.452240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.451798 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" event={"ID":"d44f04d0-36f2-48f1-a806-16bad817a5da","Type":"ContainerStarted","Data":"85cfeb00c6b92a279b979e1c32baa1ac2e754e0c728fe6fcbf6ca7a61b5a8588"} Nov 01 14:28:25.452240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.451831 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:25.453374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.453346 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" event={"ID":"ef693117-223e-41d2-b4ae-15b153d2580c","Type":"ContainerStarted","Data":"87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438"} Nov 01 14:28:25.453482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.453387 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" event={"ID":"ef693117-223e-41d2-b4ae-15b153d2580c","Type":"ContainerStarted","Data":"c5a6fcb6997a164ca509b42fc5f66e31adbc358799c6aa3cfd877f71b90b51f3"} Nov 01 14:28:25.453482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.453417 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:25.566172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.566114 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" podStartSLOduration=1.566101228 podStartE2EDuration="1.566101228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:25.565045273 +0000 UTC m=+644.623197133" watchObservedRunningTime="2026-11-01 14:28:25.566101228 +0000 UTC m=+644.624253078" Nov 01 14:28:25.683723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:25.683655 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" podStartSLOduration=2.683635858 podStartE2EDuration="2.683635858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:25.683494233 +0000 UTC m=+644.741646082" watchObservedRunningTime="2026-11-01 14:28:25.683635858 +0000 UTC m=+644.741787709" Nov 01 14:28:27.419617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:27.419581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:28.409036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:28.409008 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:28:28.409255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:28.409072 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:28:28.409255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:28.409208 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:28:31.106574 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:28:31.106541 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:28:32.012983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:32.012953 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:32.179078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:32.178994 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:32.339264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:32.339121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:36.465337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.463452 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:28:36.465337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.464491 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:28:36.516491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.516460 5473 generic.go:356] "Generic (PLEG): container finished" podID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerID="af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595" exitCode=0 Nov 01 14:28:36.516689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.516530 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" event={"ID":"0f8eb5a7-15aa-44f1-8738-a34081db3f80","Type":"ContainerDied","Data":"af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595"} Nov 01 14:28:36.516910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.516895 5473 scope.go:122] "RemoveContainer" containerID="af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595" Nov 01 14:28:36.518143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.518118 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:36.518208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.518183 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:36.518269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.518230 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:36.618315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:36.618283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:37.524189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:37.524152 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" event={"ID":"0f8eb5a7-15aa-44f1-8738-a34081db3f80","Type":"ContainerStarted","Data":"01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713"} Nov 01 14:28:37.875190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:37.874706 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:37.875190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:37.874831 5473 patch_prober.go:29] interesting pod/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 14:28:37.875190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:37.874875 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 14:28:38.876369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:38.876327 5473 patch_prober.go:29] interesting pod/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 14:28:38.876888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:38.876391 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 14:28:39.871730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:39.871585 5473 patch_prober.go:29] interesting pod/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 14:28:39.879273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:39.872757 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 14:28:40.874242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:40.873611 5473 patch_prober.go:29] interesting pod/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 14:28:40.874242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:40.873713 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 14:28:41.154879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:41.154788 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:41.871608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:41.871570 5473 patch_prober.go:29] interesting pod/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 01 14:28:41.871917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:41.871884 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 01 14:28:42.873188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:42.873161 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:42.873648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:42.873463 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:42.874203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:42.874181 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:28:44.334994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.334954 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf"] Nov 01 14:28:44.340879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.340862 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.390636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.390604 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf"] Nov 01 14:28:44.438175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.438146 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:28:44.438354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.438276 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:28:44.442359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.442337 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.442469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.442399 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.442560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.442538 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x8h2\" (UniqueName: \"kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.543852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.543812 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x8h2\" (UniqueName: \"kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.544017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.543889 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.544081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.544060 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.544687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.544653 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.546341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.546320 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.600396 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.600332 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x8h2\" (UniqueName: \"kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2\") pod \"maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:44.655567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:44.655537 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:45.024852 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:45.024822 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb543edc5_8e5f_4a8e_9331_b66d92654b17.slice/crio-5b611ea5a9aa962dc53d72bbdf6c19d687ec852928366d51d7560cb296067a55 WatchSource:0}: Error finding container 5b611ea5a9aa962dc53d72bbdf6c19d687ec852928366d51d7560cb296067a55: Status 404 returned error can't find the container with id 5b611ea5a9aa962dc53d72bbdf6c19d687ec852928366d51d7560cb296067a55 Nov 01 14:28:45.026658 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:45.026635 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf"] Nov 01 14:28:45.613334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:45.613299 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" event={"ID":"b543edc5-8e5f-4a8e-9331-b66d92654b17","Type":"ContainerStarted","Data":"5b611ea5a9aa962dc53d72bbdf6c19d687ec852928366d51d7560cb296067a55"} Nov 01 14:28:45.821674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:45.821643 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:46.163395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.163357 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:46.359438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.359335 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:46.620333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.620253 5473 generic.go:356] "Generic (PLEG): container finished" podID="7de45ce9-12db-4799-9d61-525388e0a069" containerID="e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2" exitCode=0 Nov 01 14:28:46.620691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.620326 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" event={"ID":"7de45ce9-12db-4799-9d61-525388e0a069","Type":"ContainerDied","Data":"e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2"} Nov 01 14:28:46.620691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.620670 5473 scope.go:122] "RemoveContainer" containerID="e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2" Nov 01 14:28:46.621909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.621882 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:46.621996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.621946 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:46.621996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:46.621979 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:47.652502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:47.652465 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" event={"ID":"7de45ce9-12db-4799-9d61-525388e0a069","Type":"ContainerStarted","Data":"54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07"} Nov 01 14:28:47.844677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:47.844637 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:47.845096 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:47.844977 5473 patch_prober.go:29] interesting pod/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 14:28:47.845276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:47.845251 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 14:28:48.849244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:48.845086 5473 patch_prober.go:29] interesting pod/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 14:28:48.849244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:48.845157 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 14:28:49.833248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:49.832640 5473 patch_prober.go:29] interesting pod/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 14:28:49.833248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:49.832722 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 14:28:50.389751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:50.389720 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:50.839133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:50.838658 5473 patch_prober.go:29] interesting pod/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 14:28:50.839133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:50.838728 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 14:28:51.176204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.176118 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg"] Nov 01 14:28:51.185844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.185798 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.217916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.217670 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.217916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.217755 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cvxw\" (UniqueName: \"kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.217916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.217796 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.243240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.233745 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg"] Nov 01 14:28:51.318653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.318617 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cvxw\" (UniqueName: \"kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.318893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.318876 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.319068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.319054 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.319768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.319745 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.321458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.321404 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.379649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.379609 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cvxw\" (UniqueName: \"kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw\") pod \"maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.499993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.499906 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:51.875253 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.854339 5473 patch_prober.go:29] interesting pod/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Nov 01 14:28:51.875253 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.854409 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Nov 01 14:28:51.970943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:51.970702 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg"] Nov 01 14:28:52.696781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:52.696740 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" event={"ID":"b543edc5-8e5f-4a8e-9331-b66d92654b17","Type":"ContainerStarted","Data":"403ba6c790003740e8e932eadd2c0e0662147b691592e513a6a25e50103a182b"} Nov 01 14:28:52.698190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:52.698155 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" event={"ID":"39b1608c-613d-4d55-b62f-1efb3e2cb0c4","Type":"ContainerStarted","Data":"3e481f839f103c30b088098c528e656def7a68b5bdf7cdfa485d8b64bb7d8219"} Nov 01 14:28:52.813373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:52.813316 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" podStartSLOduration=1.563653812 podStartE2EDuration="8.813302715s" totalImagesPullingTime="7.249648903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:28:52.81085159 +0000 UTC m=+671.869003437" watchObservedRunningTime="2026-11-01 14:28:52.813302715 +0000 UTC m=+671.871454565" Nov 01 14:28:52.854864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:52.854769 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:52.855602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:52.855579 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:52.856187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:52.856171 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:28:53.705291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.705181 5473 generic.go:356] "Generic (PLEG): container finished" podID="b543edc5-8e5f-4a8e-9331-b66d92654b17" containerID="403ba6c790003740e8e932eadd2c0e0662147b691592e513a6a25e50103a182b" exitCode=0 Nov 01 14:28:53.705291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.705269 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" event={"ID":"b543edc5-8e5f-4a8e-9331-b66d92654b17","Type":"ContainerDied","Data":"403ba6c790003740e8e932eadd2c0e0662147b691592e513a6a25e50103a182b"} Nov 01 14:28:53.853164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.853124 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq"] Nov 01 14:28:53.857025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.856999 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:53.922846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.922814 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq"] Nov 01 14:28:53.992973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.992886 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:53.993126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.993003 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6qd\" (UniqueName: \"kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:53.993126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:53.993063 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.094403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.094365 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.094598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.094464 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6qd\" (UniqueName: \"kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.094598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.094514 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.095081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.095060 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.096920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.096897 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.151867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.151832 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6qd\" (UniqueName: \"kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd\") pod \"maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.172684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.172662 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:54.561318 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:28:54.561273 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd996c85a_7039_4b71_b531_62dc9cd730a5.slice/crio-18fadb2aca6744523b353d4cb48ffee949b5d9b734d7821e48a0a054ca73655b WatchSource:0}: Error finding container 18fadb2aca6744523b353d4cb48ffee949b5d9b734d7821e48a0a054ca73655b: Status 404 returned error can't find the container with id 18fadb2aca6744523b353d4cb48ffee949b5d9b734d7821e48a0a054ca73655b Nov 01 14:28:54.562844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.562813 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq"] Nov 01 14:28:54.712530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.712495 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" event={"ID":"d996c85a-7039-4b71-b531-62dc9cd730a5","Type":"ContainerStarted","Data":"18fadb2aca6744523b353d4cb48ffee949b5d9b734d7821e48a0a054ca73655b"} Nov 01 14:28:54.861252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.861101 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:54.898413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:54.898387 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:55.002861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.002835 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca\") pod \"b543edc5-8e5f-4a8e-9331-b66d92654b17\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " Nov 01 14:28:55.002993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.002941 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca\") pod \"b543edc5-8e5f-4a8e-9331-b66d92654b17\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " Nov 01 14:28:55.003064 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.003050 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2\" (UniqueName: \"kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2\") pod \"b543edc5-8e5f-4a8e-9331-b66d92654b17\" (UID: \"b543edc5-8e5f-4a8e-9331-b66d92654b17\") " Nov 01 14:28:55.003280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.003258 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca" pod "b543edc5-8e5f-4a8e-9331-b66d92654b17" (UID: "b543edc5-8e5f-4a8e-9331-b66d92654b17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:28:55.003416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.003401 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:55.004982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.004954 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca" pod "b543edc5-8e5f-4a8e-9331-b66d92654b17" (UID: "b543edc5-8e5f-4a8e-9331-b66d92654b17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:28:55.005069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.005044 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2" pod "b543edc5-8e5f-4a8e-9331-b66d92654b17" (UID: "b543edc5-8e5f-4a8e-9331-b66d92654b17"). InnerVolumeSpecName "kube-api-access-9x8h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:28:55.104302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.104263 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b543edc5-8e5f-4a8e-9331-b66d92654b17-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:55.104302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.104302 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x8h2\" (UniqueName: \"kubernetes.io/projected/b543edc5-8e5f-4a8e-9331-b66d92654b17-kube-api-access-9x8h2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:55.720702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.720611 5473 generic.go:356] "Generic (PLEG): container finished" podID="d996c85a-7039-4b71-b531-62dc9cd730a5" containerID="2d2d0fca32d2e3125fbe803ae9f943388c27d6eceabab56674e4396ebd4568af" exitCode=0 Nov 01 14:28:55.720702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.720687 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" event={"ID":"d996c85a-7039-4b71-b531-62dc9cd730a5","Type":"ContainerDied","Data":"2d2d0fca32d2e3125fbe803ae9f943388c27d6eceabab56674e4396ebd4568af"} Nov 01 14:28:55.722075 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.722059 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" Nov 01 14:28:55.722075 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.722070 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf" event={"ID":"b543edc5-8e5f-4a8e-9331-b66d92654b17","Type":"ContainerDied","Data":"5b611ea5a9aa962dc53d72bbdf6c19d687ec852928366d51d7560cb296067a55"} Nov 01 14:28:55.722275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.722100 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b611ea5a9aa962dc53d72bbdf6c19d687ec852928366d51d7560cb296067a55" Nov 01 14:28:55.723872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.723851 5473 generic.go:356] "Generic (PLEG): container finished" podID="39b1608c-613d-4d55-b62f-1efb3e2cb0c4" containerID="24df77910c2fc699a61daa711d542d87c94d2e074dbc073739fec33e1b07900f" exitCode=0 Nov 01 14:28:55.723988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:55.723879 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" event={"ID":"39b1608c-613d-4d55-b62f-1efb3e2cb0c4","Type":"ContainerDied","Data":"24df77910c2fc699a61daa711d542d87c94d2e074dbc073739fec33e1b07900f"} Nov 01 14:28:56.895043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:56.895013 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:56.898609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:56.898582 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:57.024970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.024942 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca\") pod \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " Nov 01 14:28:57.025164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.024977 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca\") pod \"d996c85a-7039-4b71-b531-62dc9cd730a5\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " Nov 01 14:28:57.025164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.025005 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw\" (UniqueName: \"kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw\") pod \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " Nov 01 14:28:57.025164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.025048 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca\") pod \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\" (UID: \"39b1608c-613d-4d55-b62f-1efb3e2cb0c4\") " Nov 01 14:28:57.025164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.025077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca\") pod \"d996c85a-7039-4b71-b531-62dc9cd730a5\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " Nov 01 14:28:57.025408 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.025191 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd\" (UniqueName: \"kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd\") pod \"d996c85a-7039-4b71-b531-62dc9cd730a5\" (UID: \"d996c85a-7039-4b71-b531-62dc9cd730a5\") " Nov 01 14:28:57.025651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.025623 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca" pod "d996c85a-7039-4b71-b531-62dc9cd730a5" (UID: "d996c85a-7039-4b71-b531-62dc9cd730a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:28:57.025867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.025839 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca" pod "39b1608c-613d-4d55-b62f-1efb3e2cb0c4" (UID: "39b1608c-613d-4d55-b62f-1efb3e2cb0c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:28:57.027256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.027230 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca" pod "d996c85a-7039-4b71-b531-62dc9cd730a5" (UID: "d996c85a-7039-4b71-b531-62dc9cd730a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:28:57.027672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.027651 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw" pod "39b1608c-613d-4d55-b62f-1efb3e2cb0c4" (UID: "39b1608c-613d-4d55-b62f-1efb3e2cb0c4"). InnerVolumeSpecName "kube-api-access-9cvxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:28:57.027766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.027724 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca" pod "39b1608c-613d-4d55-b62f-1efb3e2cb0c4" (UID: "39b1608c-613d-4d55-b62f-1efb3e2cb0c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:28:57.027831 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.027767 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd" pod "d996c85a-7039-4b71-b531-62dc9cd730a5" (UID: "d996c85a-7039-4b71-b531-62dc9cd730a5"). InnerVolumeSpecName "kube-api-access-mr6qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:28:57.126977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.126940 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:57.126977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.126980 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:57.127156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.126997 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cvxw\" (UniqueName: \"kubernetes.io/projected/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-kube-api-access-9cvxw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:57.127156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.127012 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39b1608c-613d-4d55-b62f-1efb3e2cb0c4-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:57.127156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.127027 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d996c85a-7039-4b71-b531-62dc9cd730a5-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:57.127156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.127042 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6qd\" (UniqueName: \"kubernetes.io/projected/d996c85a-7039-4b71-b531-62dc9cd730a5-kube-api-access-mr6qd\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:28:57.736795 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.736763 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" Nov 01 14:28:57.736971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.736795 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg" event={"ID":"39b1608c-613d-4d55-b62f-1efb3e2cb0c4","Type":"ContainerDied","Data":"3e481f839f103c30b088098c528e656def7a68b5bdf7cdfa485d8b64bb7d8219"} Nov 01 14:28:57.736971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.736860 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e481f839f103c30b088098c528e656def7a68b5bdf7cdfa485d8b64bb7d8219" Nov 01 14:28:57.738388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.738363 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" event={"ID":"d996c85a-7039-4b71-b531-62dc9cd730a5","Type":"ContainerDied","Data":"18fadb2aca6744523b353d4cb48ffee949b5d9b734d7821e48a0a054ca73655b"} Nov 01 14:28:57.738388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.738379 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq" Nov 01 14:28:57.738540 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:57.738404 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18fadb2aca6744523b353d4cb48ffee949b5d9b734d7821e48a0a054ca73655b" Nov 01 14:28:58.745839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:58.745764 5473 generic.go:356] "Generic (PLEG): container finished" podID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerID="1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8" exitCode=0 Nov 01 14:28:58.746248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:58.745838 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" event={"ID":"b0533b77-9087-4fde-a8cf-f3481223bbd9","Type":"ContainerDied","Data":"1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8"} Nov 01 14:28:58.746248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:58.746204 5473 scope.go:122] "RemoveContainer" containerID="1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8" Nov 01 14:28:58.747353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:58.747329 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:58.747418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:58.747394 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:58.747452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:58.747424 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441198 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624"] Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441840 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d996c85a-7039-4b71-b531-62dc9cd730a5" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441857 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d996c85a-7039-4b71-b531-62dc9cd730a5" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441890 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b543edc5-8e5f-4a8e-9331-b66d92654b17" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441899 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b543edc5-8e5f-4a8e-9331-b66d92654b17" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441916 5473 cpu_manager.go:414] "containerMap: removing container" podUID="39b1608c-613d-4d55-b62f-1efb3e2cb0c4" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.441924 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b1608c-613d-4d55-b62f-1efb3e2cb0c4" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.442040 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d996c85a-7039-4b71-b531-62dc9cd730a5" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.442055 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b543edc5-8e5f-4a8e-9331-b66d92654b17" containerName="revoke-keys" Nov 01 14:28:59.443239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.442073 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b1608c-613d-4d55-b62f-1efb3e2cb0c4" containerName="revoke-keys" Nov 01 14:28:59.452034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.452006 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.491254 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.491205 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624"] Nov 01 14:28:59.504672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.504646 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:28:59.504870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.504850 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:28:59.552946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.552909 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.553150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.552961 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.553150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.552981 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9m4\" (UniqueName: \"kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.654463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.654421 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.654463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.654468 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.654729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.654498 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9m4\" (UniqueName: \"kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.655154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.655125 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.656814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.656794 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.713839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.713765 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9m4\" (UniqueName: \"kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4\") pod \"maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.756549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.756513 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" event={"ID":"b0533b77-9087-4fde-a8cf-f3481223bbd9","Type":"ContainerStarted","Data":"8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86"} Nov 01 14:28:59.768013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.767992 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:28:59.831715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.828766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:28:59.988510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.988433 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:28:59.989412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.989390 5473 patch_prober.go:29] interesting pod/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 14:28:59.989470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:28:59.989438 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 14:29:00.149723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:00.149685 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624"] Nov 01 14:29:00.150350 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:00.150324 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64886668_e326_4441_adb8_158c60d81009.slice/crio-c0bbe1b860f6032fc718d5c9bcc7c9995a54953860045f17197283188336a6d0 WatchSource:0}: Error finding container c0bbe1b860f6032fc718d5c9bcc7c9995a54953860045f17197283188336a6d0: Status 404 returned error can't find the container with id c0bbe1b860f6032fc718d5c9bcc7c9995a54953860045f17197283188336a6d0 Nov 01 14:29:00.763364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:00.763331 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" event={"ID":"64886668-e326-4441-adb8-158c60d81009","Type":"ContainerStarted","Data":"808c030a2ca3a4a61162129cae649ac038de1c0e9b2bed4788c970c418421460"} Nov 01 14:29:00.763822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:00.763390 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" event={"ID":"64886668-e326-4441-adb8-158c60d81009","Type":"ContainerStarted","Data":"c0bbe1b860f6032fc718d5c9bcc7c9995a54953860045f17197283188336a6d0"} Nov 01 14:29:00.883488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:00.883434 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" podStartSLOduration=1.883420495 podStartE2EDuration="1.883420495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:00.882062589 +0000 UTC m=+679.940214439" watchObservedRunningTime="2026-11-01 14:29:00.883420495 +0000 UTC m=+679.941572402" Nov 01 14:29:00.989022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:00.988986 5473 patch_prober.go:29] interesting pod/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 14:29:00.989193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:00.989046 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 14:29:01.107474 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:01.107384 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:29:01.289080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.289049 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:29:01.770065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.770029 5473 generic.go:356] "Generic (PLEG): container finished" podID="64886668-e326-4441-adb8-158c60d81009" containerID="808c030a2ca3a4a61162129cae649ac038de1c0e9b2bed4788c970c418421460" exitCode=0 Nov 01 14:29:01.770457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.770106 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" event={"ID":"64886668-e326-4441-adb8-158c60d81009","Type":"ContainerDied","Data":"808c030a2ca3a4a61162129cae649ac038de1c0e9b2bed4788c970c418421460"} Nov 01 14:29:01.881451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.881419 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j"] Nov 01 14:29:01.984494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.984461 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm"] Nov 01 14:29:01.988427 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.988402 5473 patch_prober.go:29] interesting pod/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 14:29:01.988558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:01.988452 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 14:29:02.310450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:02.310421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:02.506487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:02.506453 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:02.958988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:02.958962 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:29:02.988322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:02.988293 5473 patch_prober.go:29] interesting pod/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 14:29:02.988462 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:02.988343 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 14:29:03.095963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.095886 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca\") pod \"64886668-e326-4441-adb8-158c60d81009\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " Nov 01 14:29:03.096143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.096035 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4\" (UniqueName: \"kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4\") pod \"64886668-e326-4441-adb8-158c60d81009\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " Nov 01 14:29:03.096143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.096098 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca\") pod \"64886668-e326-4441-adb8-158c60d81009\" (UID: \"64886668-e326-4441-adb8-158c60d81009\") " Nov 01 14:29:03.096377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.096347 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca" pod "64886668-e326-4441-adb8-158c60d81009" (UID: "64886668-e326-4441-adb8-158c60d81009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:03.096470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.096453 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64886668-e326-4441-adb8-158c60d81009-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:03.098161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.098132 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4" pod "64886668-e326-4441-adb8-158c60d81009" (UID: "64886668-e326-4441-adb8-158c60d81009"). InnerVolumeSpecName "kube-api-access-zf9m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:03.098355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.098323 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca" pod "64886668-e326-4441-adb8-158c60d81009" (UID: "64886668-e326-4441-adb8-158c60d81009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:29:03.197756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.197719 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf9m4\" (UniqueName: \"kubernetes.io/projected/64886668-e326-4441-adb8-158c60d81009-kube-api-access-zf9m4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:03.197756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.197747 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64886668-e326-4441-adb8-158c60d81009-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:03.782062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.782018 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" event={"ID":"64886668-e326-4441-adb8-158c60d81009","Type":"ContainerDied","Data":"c0bbe1b860f6032fc718d5c9bcc7c9995a54953860045f17197283188336a6d0"} Nov 01 14:29:03.782062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.782054 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624" Nov 01 14:29:03.782299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.782112 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0bbe1b860f6032fc718d5c9bcc7c9995a54953860045f17197283188336a6d0" Nov 01 14:29:03.988705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.988672 5473 patch_prober.go:29] interesting pod/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 01 14:29:03.989129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:03.988727 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 01 14:29:04.661643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:04.661609 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:04.992323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:04.992236 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:29:04.993292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:04.993274 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:29:05.250185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.250103 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h"] Nov 01 14:29:05.250606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.250592 5473 cpu_manager.go:414] "containerMap: removing container" podUID="64886668-e326-4441-adb8-158c60d81009" containerName="revoke-keys" Nov 01 14:29:05.250649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.250607 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64886668-e326-4441-adb8-158c60d81009" containerName="revoke-keys" Nov 01 14:29:05.250684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.250669 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="64886668-e326-4441-adb8-158c60d81009" containerName="revoke-keys" Nov 01 14:29:05.253506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.253492 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.309043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.309011 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:29:05.309268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.309250 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:29:05.312604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.312576 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h"] Nov 01 14:29:05.321905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.321875 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.322029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.321939 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.322103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.322039 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwzb\" (UniqueName: \"kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.422765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.422727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.422976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.422789 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.422976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.422822 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwzb\" (UniqueName: \"kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.423478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.423457 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.425208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.425189 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.480336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.480304 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwzb\" (UniqueName: \"kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb\") pod \"maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.566382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.566350 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:05.827961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.827876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:05.935265 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:05.935235 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142d0d71_364a_4929_b3c0_5089e164982a.slice/crio-73d8175a67fde7e2091c790e84702202cc89d5a415dfdaf74ac7fb86ab788d7a WatchSource:0}: Error finding container 73d8175a67fde7e2091c790e84702202cc89d5a415dfdaf74ac7fb86ab788d7a: Status 404 returned error can't find the container with id 73d8175a67fde7e2091c790e84702202cc89d5a415dfdaf74ac7fb86ab788d7a Nov 01 14:29:05.938137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:05.938110 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h"] Nov 01 14:29:06.006399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:06.006368 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:06.802131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:06.802098 5473 generic.go:356] "Generic (PLEG): container finished" podID="142d0d71-364a-4929-b3c0-5089e164982a" containerID="20cb51a07698200f011601e1544adcc7ddf5f485e66188b5b648c3ae743e0312" exitCode=0 Nov 01 14:29:06.802131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:06.802173 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" event={"ID":"142d0d71-364a-4929-b3c0-5089e164982a","Type":"ContainerDied","Data":"20cb51a07698200f011601e1544adcc7ddf5f485e66188b5b648c3ae743e0312"} Nov 01 14:29:06.802131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:06.802244 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" event={"ID":"142d0d71-364a-4929-b3c0-5089e164982a","Type":"ContainerStarted","Data":"73d8175a67fde7e2091c790e84702202cc89d5a415dfdaf74ac7fb86ab788d7a"} Nov 01 14:29:06.882369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:06.882302 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" podUID="48c4fb1b-ff2e-4ab6-a704-0203bc438439" containerName="payload-processing" containerID="cri-o://8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a" gracePeriod=25 Nov 01 14:29:06.984823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:06.984773 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" podUID="e6ddd089-3c38-4334-bce3-6722f2d7a216" containerName="payload-pre-processing" containerID="cri-o://6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc" gracePeriod=25 Nov 01 14:29:07.139890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.139866 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:29:07.237763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.237739 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:29:07.241564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.241547 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume\") pod \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " Nov 01 14:29:07.241661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.241637 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz\" (UniqueName: \"kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz\") pod \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\" (UID: \"48c4fb1b-ff2e-4ab6-a704-0203bc438439\") " Nov 01 14:29:07.241960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.241932 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume" pod "48c4fb1b-ff2e-4ab6-a704-0203bc438439" (UID: "48c4fb1b-ff2e-4ab6-a704-0203bc438439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:07.243654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.243635 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz" pod "48c4fb1b-ff2e-4ab6-a704-0203bc438439" (UID: "48c4fb1b-ff2e-4ab6-a704-0203bc438439"). InnerVolumeSpecName "kube-api-access-pftrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:07.342725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.342637 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume\") pod \"e6ddd089-3c38-4334-bce3-6722f2d7a216\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " Nov 01 14:29:07.342725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.342710 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b\" (UniqueName: \"kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b\") pod \"e6ddd089-3c38-4334-bce3-6722f2d7a216\" (UID: \"e6ddd089-3c38-4334-bce3-6722f2d7a216\") " Nov 01 14:29:07.342946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.342933 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftrz\" (UniqueName: \"kubernetes.io/projected/48c4fb1b-ff2e-4ab6-a704-0203bc438439-kube-api-access-pftrz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:07.342985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.342947 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48c4fb1b-ff2e-4ab6-a704-0203bc438439-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:07.343036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.343012 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume" pod "e6ddd089-3c38-4334-bce3-6722f2d7a216" (UID: "e6ddd089-3c38-4334-bce3-6722f2d7a216"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:07.344758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.344739 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b" pod "e6ddd089-3c38-4334-bce3-6722f2d7a216" (UID: "e6ddd089-3c38-4334-bce3-6722f2d7a216"). InnerVolumeSpecName "kube-api-access-cx26b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:07.443804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.443774 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ddd089-3c38-4334-bce3-6722f2d7a216-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:07.443804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.443798 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx26b\" (UniqueName: \"kubernetes.io/projected/e6ddd089-3c38-4334-bce3-6722f2d7a216-kube-api-access-cx26b\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:07.810354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.810325 5473 generic.go:356] "Generic (PLEG): container finished" podID="e6ddd089-3c38-4334-bce3-6722f2d7a216" containerID="6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc" exitCode=0 Nov 01 14:29:07.810536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.810373 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" Nov 01 14:29:07.810536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.810402 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" event={"ID":"e6ddd089-3c38-4334-bce3-6722f2d7a216","Type":"ContainerDied","Data":"6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc"} Nov 01 14:29:07.810536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.810463 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm" event={"ID":"e6ddd089-3c38-4334-bce3-6722f2d7a216","Type":"ContainerDied","Data":"bb2c1d05d7b759e14f0660b9999c5a4f1e4dd35f952ee11205513caeb13c6f74"} Nov 01 14:29:07.810536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.810491 5473 scope.go:122] "RemoveContainer" containerID="6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc" Nov 01 14:29:07.811695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.811676 5473 generic.go:356] "Generic (PLEG): container finished" podID="48c4fb1b-ff2e-4ab6-a704-0203bc438439" containerID="8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a" exitCode=0 Nov 01 14:29:07.811750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.811731 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" Nov 01 14:29:07.811799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.811780 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" event={"ID":"48c4fb1b-ff2e-4ab6-a704-0203bc438439","Type":"ContainerDied","Data":"8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a"} Nov 01 14:29:07.811850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.811819 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j" event={"ID":"48c4fb1b-ff2e-4ab6-a704-0203bc438439","Type":"ContainerDied","Data":"f893cd6ed4cd171bd8986b7fb35923296c5fccefafb8020d28e921506c086757"} Nov 01 14:29:07.821120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.821100 5473 scope.go:122] "RemoveContainer" containerID="6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc" Nov 01 14:29:07.821535 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:07.821445 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc\": container with ID starting with 6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc not found: ID does not exist" containerID="6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc" Nov 01 14:29:07.821535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.821478 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc"} err="failed to get container status \"6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc\": rpc error: code = NotFound desc = could not find container \"6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc\": container with ID starting with 6a570aed45cc0ee2f553156137fc025f60c17a58ffc70eedacbd4a5e11706cdc not found: ID does not exist" Nov 01 14:29:07.821535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.821496 5473 scope.go:122] "RemoveContainer" containerID="8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a" Nov 01 14:29:07.831176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.831158 5473 scope.go:122] "RemoveContainer" containerID="8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a" Nov 01 14:29:07.831502 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:07.831482 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a\": container with ID starting with 8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a not found: ID does not exist" containerID="8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a" Nov 01 14:29:07.831585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.831511 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a"} err="failed to get container status \"8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a\": rpc error: code = NotFound desc = could not find container \"8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a\": container with ID starting with 8b74b7ff8755ab9a9c8ac33ffd3f196e734f96ffebd91f0efb7423255c03c74a not found: ID does not exist" Nov 01 14:29:07.947068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.947034 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:07.975052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.975025 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm"] Nov 01 14:29:07.977267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:07.977248 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d8f1c66f-5868bff47f-57wgm"] Nov 01 14:29:08.048448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.048418 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca\") pod \"142d0d71-364a-4929-b3c0-5089e164982a\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " Nov 01 14:29:08.048612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.048461 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca\") pod \"142d0d71-364a-4929-b3c0-5089e164982a\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " Nov 01 14:29:08.048612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.048560 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb\" (UniqueName: \"kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb\") pod \"142d0d71-364a-4929-b3c0-5089e164982a\" (UID: \"142d0d71-364a-4929-b3c0-5089e164982a\") " Nov 01 14:29:08.048847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.048821 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca" pod "142d0d71-364a-4929-b3c0-5089e164982a" (UID: "142d0d71-364a-4929-b3c0-5089e164982a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:08.050611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.050584 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb" pod "142d0d71-364a-4929-b3c0-5089e164982a" (UID: "142d0d71-364a-4929-b3c0-5089e164982a"). InnerVolumeSpecName "kube-api-access-gxwzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:08.050709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.050613 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca" pod "142d0d71-364a-4929-b3c0-5089e164982a" (UID: "142d0d71-364a-4929-b3c0-5089e164982a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:29:08.087992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.087913 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j"] Nov 01 14:29:08.090151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.090127 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d8f1c66f-7c4f886c67-cfq6j"] Nov 01 14:29:08.110574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.110549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:08.149695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.149665 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxwzb\" (UniqueName: \"kubernetes.io/projected/142d0d71-364a-4929-b3c0-5089e164982a-kube-api-access-gxwzb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:08.149695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.149692 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:08.150102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.149704 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142d0d71-364a-4929-b3c0-5089e164982a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:08.311389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.311356 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:08.822164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.822127 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" event={"ID":"142d0d71-364a-4929-b3c0-5089e164982a","Type":"ContainerDied","Data":"73d8175a67fde7e2091c790e84702202cc89d5a415dfdaf74ac7fb86ab788d7a"} Nov 01 14:29:08.822164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.822160 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h" Nov 01 14:29:08.822164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.822185 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73d8175a67fde7e2091c790e84702202cc89d5a415dfdaf74ac7fb86ab788d7a" Nov 01 14:29:08.906945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:08.906913 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:09.080114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:09.080026 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c4fb1b-ff2e-4ab6-a704-0203bc438439" path="/var/lib/kubelet/pods/48c4fb1b-ff2e-4ab6-a704-0203bc438439/volumes" Nov 01 14:29:09.080490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:09.080470 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ddd089-3c38-4334-bce3-6722f2d7a216" path="/var/lib/kubelet/pods/e6ddd089-3c38-4334-bce3-6722f2d7a216/volumes" Nov 01 14:29:10.055879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.055843 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:10.114737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.114710 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq"] Nov 01 14:29:10.380526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.380446 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:10.595703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.595675 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:10.768775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.768744 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:10.864964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.864926 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg"] Nov 01 14:29:10.869345 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:10.869324 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d8f1c66f-6d4bg"] Nov 01 14:29:11.096402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:11.096321 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b1608c-613d-4d55-b62f-1efb3e2cb0c4" path="/var/lib/kubelet/pods/39b1608c-613d-4d55-b62f-1efb3e2cb0c4/volumes" Nov 01 14:29:11.172958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:11.172921 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:11.476401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:11.476328 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:11.662745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:11.662712 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:11.931041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:11.931012 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:11.993103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:11.993076 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq"] Nov 01 14:29:12.043284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:12.043246 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e27521d-8ktqq"] Nov 01 14:29:12.059106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:12.059068 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd"] Nov 01 14:29:12.059354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:12.059322 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" containerID="cri-o://8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86" gracePeriod=30 Nov 01 14:29:12.136883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:12.136848 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:12.301919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:12.301891 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:12.825013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:12.824982 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:13.000527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:13.000495 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:13.081321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:13.081245 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d996c85a-7039-4b71-b531-62dc9cd730a5" path="/var/lib/kubelet/pods/d996c85a-7039-4b71-b531-62dc9cd730a5/volumes" Nov 01 14:29:13.167526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:13.167496 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:13.634732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:13.634700 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:13.832111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:13.832073 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:15.115764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.115706 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" podUID="70e68d61-32a5-4ffe-987f-8a82e3181db0" containerName="payload-pre-processing" containerID="cri-o://cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d" gracePeriod=25 Nov 01 14:29:15.365563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.365540 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:29:15.433478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.433397 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume\") pod \"70e68d61-32a5-4ffe-987f-8a82e3181db0\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " Nov 01 14:29:15.433635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.433493 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995\" (UniqueName: \"kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995\") pod \"70e68d61-32a5-4ffe-987f-8a82e3181db0\" (UID: \"70e68d61-32a5-4ffe-987f-8a82e3181db0\") " Nov 01 14:29:15.433833 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.433806 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume" pod "70e68d61-32a5-4ffe-987f-8a82e3181db0" (UID: "70e68d61-32a5-4ffe-987f-8a82e3181db0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:15.433913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.433896 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e68d61-32a5-4ffe-987f-8a82e3181db0-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:15.435664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.435641 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995" pod "70e68d61-32a5-4ffe-987f-8a82e3181db0" (UID: "70e68d61-32a5-4ffe-987f-8a82e3181db0"). InnerVolumeSpecName "kube-api-access-98995". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:15.535425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.535387 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98995\" (UniqueName: \"kubernetes.io/projected/70e68d61-32a5-4ffe-987f-8a82e3181db0-kube-api-access-98995\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:15.590922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.590868 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:15.863573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.863541 5473 generic.go:356] "Generic (PLEG): container finished" podID="70e68d61-32a5-4ffe-987f-8a82e3181db0" containerID="cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d" exitCode=0 Nov 01 14:29:15.863759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.863587 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" Nov 01 14:29:15.863759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.863623 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" event={"ID":"70e68d61-32a5-4ffe-987f-8a82e3181db0","Type":"ContainerDied","Data":"cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d"} Nov 01 14:29:15.863759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.863678 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq" event={"ID":"70e68d61-32a5-4ffe-987f-8a82e3181db0","Type":"ContainerDied","Data":"971db1bc17a4da17c4bcc09fc7a55f4ea3005eefcb6337e0955e29dcd4533f75"} Nov 01 14:29:15.863759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.863706 5473 scope.go:122] "RemoveContainer" containerID="cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d" Nov 01 14:29:15.873915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.873895 5473 scope.go:122] "RemoveContainer" containerID="cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d" Nov 01 14:29:15.874176 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:15.874153 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d\": container with ID starting with cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d not found: ID does not exist" containerID="cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d" Nov 01 14:29:15.874237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.874186 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d"} err="failed to get container status \"cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d\": rpc error: code = NotFound desc = could not find container \"cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d\": container with ID starting with cde9ebc554f59714b9f78aa60933774b9b4240d339712ef2fa1e73e18ecfac7d not found: ID does not exist" Nov 01 14:29:15.885004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:15.884982 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:16.032325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.032290 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq"] Nov 01 14:29:16.082596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.082566 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c9fe7b2a-6b97b578c5-wpldq"] Nov 01 14:29:16.084710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.084691 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624"] Nov 01 14:29:16.086686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.086662 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c9fe7b2a-82624"] Nov 01 14:29:16.117196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.117100 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd"] Nov 01 14:29:16.117696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.117398 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" containerID="cri-o://01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713" gracePeriod=30 Nov 01 14:29:16.190943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.190908 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:16.962307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:16.962273 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:17.090430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.090386 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64886668-e326-4441-adb8-158c60d81009" path="/var/lib/kubelet/pods/64886668-e326-4441-adb8-158c60d81009/volumes" Nov 01 14:29:17.091232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.091197 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e68d61-32a5-4ffe-987f-8a82e3181db0" path="/var/lib/kubelet/pods/70e68d61-32a5-4ffe-987f-8a82e3181db0/volumes" Nov 01 14:29:17.227716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.227643 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:17.317554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.317532 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:29:17.454585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454532 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.454773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454645 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.454838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454769 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.454838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454798 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.454838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454830 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.454998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454890 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.454998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.454918 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.455108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.455000 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.455108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.455030 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo\") pod \"b0533b77-9087-4fde-a8cf-f3481223bbd9\" (UID: \"b0533b77-9087-4fde-a8cf-f3481223bbd9\") " Nov 01 14:29:17.455507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.455476 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:17.456190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.455689 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:17.456190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.455873 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:17.456190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.455910 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:17.456553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.456529 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:17.462210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.462180 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:17.462480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.462459 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "kube-api-access-mzq9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:17.462786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.462765 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:17.464353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.464298 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo" pod "b0533b77-9087-4fde-a8cf-f3481223bbd9" (UID: "b0533b77-9087-4fde-a8cf-f3481223bbd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:29:17.543731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.543697 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr"] Nov 01 14:29:17.548886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.548852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:17.556439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556411 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556441 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556457 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0533b77-9087-4fde-a8cf-f3481223bbd9-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556470 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556484 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556498 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzq9r\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-kube-api-access-mzq9r\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556512 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556525 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.556560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.556538 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0533b77-9087-4fde-a8cf-f3481223bbd9-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:17.603313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.603278 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw"] Nov 01 14:29:17.745868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.745832 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:17.879392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.879306 5473 generic.go:356] "Generic (PLEG): container finished" podID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerID="8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86" exitCode=0 Nov 01 14:29:17.879392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.879360 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" Nov 01 14:29:17.879581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.879386 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" event={"ID":"b0533b77-9087-4fde-a8cf-f3481223bbd9","Type":"ContainerDied","Data":"8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86"} Nov 01 14:29:17.879581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.879437 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd" event={"ID":"b0533b77-9087-4fde-a8cf-f3481223bbd9","Type":"ContainerDied","Data":"74e422591635177e527b4bdb92641899f42c86ae4998348f57a3d371dab29757"} Nov 01 14:29:17.879581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.879459 5473 scope.go:122] "RemoveContainer" containerID="8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86" Nov 01 14:29:17.889196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.889176 5473 scope.go:122] "RemoveContainer" containerID="1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8" Nov 01 14:29:17.899362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.899344 5473 scope.go:122] "RemoveContainer" containerID="8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86" Nov 01 14:29:17.899679 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:17.899655 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86\": container with ID starting with 8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86 not found: ID does not exist" containerID="8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86" Nov 01 14:29:17.899766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.899686 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86"} err="failed to get container status \"8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86\": rpc error: code = NotFound desc = could not find container \"8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86\": container with ID starting with 8154a505249418fe361c3e2111e7a3d03c3a07ea5299fc3ae6d19e9fb2907b86 not found: ID does not exist" Nov 01 14:29:17.899766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.899704 5473 scope.go:122] "RemoveContainer" containerID="1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8" Nov 01 14:29:17.899991 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:17.899973 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8\": container with ID starting with 1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8 not found: ID does not exist" containerID="1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8" Nov 01 14:29:17.900084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.899994 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8"} err="failed to get container status \"1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8\": rpc error: code = NotFound desc = could not find container \"1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8\": container with ID starting with 1d67799014418f15a963b277b1fcdb47dd26170f8c9c570a7a873b11c4b740b8 not found: ID does not exist" Nov 01 14:29:17.925347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:17.925320 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:18.044105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:18.044071 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd"] Nov 01 14:29:18.047022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:18.047001 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e27521d-openshift-default-6c6555bdf-jq5vd"] Nov 01 14:29:18.100887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:18.100852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:18.300952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:18.300918 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:18.592915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:18.592838 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:19.081134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:19.081101 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" path="/var/lib/kubelet/pods/b0533b77-9087-4fde-a8cf-f3481223bbd9/volumes" Nov 01 14:29:20.802514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:20.802486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:21.045263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.045164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:21.217976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.217943 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf"] Nov 01 14:29:21.268094 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.268063 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f7c71759-2hhdf"] Nov 01 14:29:21.371109 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.371083 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:29:21.493830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493756 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.493830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493797 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.493830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493822 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493876 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493902 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493938 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.493997 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.494040 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.494079 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy\") pod \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\" (UID: \"0f8eb5a7-15aa-44f1-8738-a34081db3f80\") " Nov 01 14:29:21.494104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.494054 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:21.494737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.494349 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:21.494868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.494656 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:21.494868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.494424 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.495073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.495052 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:21.496289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.496261 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:21.496873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.496849 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:21.497112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.497085 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "kube-api-access-w97np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:21.497330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.497311 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:21.497388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.497335 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo" pod "0f8eb5a7-15aa-44f1-8738-a34081db3f80" (UID: "0f8eb5a7-15aa-44f1-8738-a34081db3f80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:29:21.596103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596073 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596098 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596108 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596117 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596127 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596136 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w97np\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-kube-api-access-w97np\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596145 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.596359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.596153 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8eb5a7-15aa-44f1-8738-a34081db3f80-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:21.775068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.775037 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:21.914410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.914381 5473 generic.go:356] "Generic (PLEG): container finished" podID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerID="01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713" exitCode=0 Nov 01 14:29:21.914823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.914421 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" event={"ID":"0f8eb5a7-15aa-44f1-8738-a34081db3f80","Type":"ContainerDied","Data":"01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713"} Nov 01 14:29:21.914823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.914433 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" Nov 01 14:29:21.914823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.914462 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd" event={"ID":"0f8eb5a7-15aa-44f1-8738-a34081db3f80","Type":"ContainerDied","Data":"ef00acf5c8e3610a705aa8f4d7a3cc3ae700036770b42601e04a1d8107472250"} Nov 01 14:29:21.914823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.914483 5473 scope.go:122] "RemoveContainer" containerID="01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713" Nov 01 14:29:21.924567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.924548 5473 scope.go:122] "RemoveContainer" containerID="af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595" Nov 01 14:29:21.933266 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.933246 5473 scope.go:122] "RemoveContainer" containerID="01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713" Nov 01 14:29:21.933530 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:21.933512 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713\": container with ID starting with 01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713 not found: ID does not exist" containerID="01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713" Nov 01 14:29:21.933569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.933541 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713"} err="failed to get container status \"01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713\": rpc error: code = NotFound desc = could not find container \"01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713\": container with ID starting with 01ab78b29fec2b99f734a54457d2606dfef6981e3690762d84bd7208dff2d713 not found: ID does not exist" Nov 01 14:29:21.933569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.933559 5473 scope.go:122] "RemoveContainer" containerID="af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595" Nov 01 14:29:21.933839 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:21.933819 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595\": container with ID starting with af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595 not found: ID does not exist" containerID="af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595" Nov 01 14:29:21.933876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:21.933846 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595"} err="failed to get container status \"af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595\": rpc error: code = NotFound desc = could not find container \"af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595\": container with ID starting with af0e154fb8e39c8757111de0e5ab782e7db22dbbb1811eb072e01aad4512a595 not found: ID does not exist" Nov 01 14:29:22.030361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.030207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:22.087602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.087572 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd"] Nov 01 14:29:22.135450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.135415 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c9fe7b2a-gw-openshift-default-77f4b7d557-q7cdd"] Nov 01 14:29:22.218645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.218619 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h"] Nov 01 14:29:22.237364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.237340 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c0f3259b-qd45h"] Nov 01 14:29:22.399430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.399352 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv"] Nov 01 14:29:22.399610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.399568 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" containerID="cri-o://54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07" gracePeriod=30 Nov 01 14:29:22.432322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.432292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:22.544662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.544607 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" podUID="d44f04d0-36f2-48f1-a806-16bad817a5da" containerName="payload-processing" containerID="cri-o://65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84" gracePeriod=25 Nov 01 14:29:22.601323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.601292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:22.604127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.604098 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" podUID="ef693117-223e-41d2-b4ae-15b153d2580c" containerName="payload-pre-processing" containerID="cri-o://87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438" gracePeriod=25 Nov 01 14:29:22.765819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.765779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:22.807190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807159 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7"] Nov 01 14:29:22.807651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807630 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807652 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807670 5473 cpu_manager.go:414] "containerMap: removing container" podUID="48c4fb1b-ff2e-4ab6-a704-0203bc438439" containerName="payload-processing" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807677 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c4fb1b-ff2e-4ab6-a704-0203bc438439" containerName="payload-processing" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807689 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807697 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807709 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" Nov 01 14:29:22.807733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807717 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807739 5473 cpu_manager.go:414] "containerMap: removing container" podUID="70e68d61-32a5-4ffe-987f-8a82e3181db0" containerName="payload-pre-processing" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807747 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e68d61-32a5-4ffe-987f-8a82e3181db0" containerName="payload-pre-processing" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807757 5473 cpu_manager.go:414] "containerMap: removing container" podUID="142d0d71-364a-4929-b3c0-5089e164982a" containerName="revoke-keys" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807765 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142d0d71-364a-4929-b3c0-5089e164982a" containerName="revoke-keys" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807779 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807786 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807814 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e6ddd089-3c38-4334-bce3-6722f2d7a216" containerName="payload-pre-processing" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807822 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ddd089-3c38-4334-bce3-6722f2d7a216" containerName="payload-pre-processing" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807918 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c4fb1b-ff2e-4ab6-a704-0203bc438439" containerName="payload-processing" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807929 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807945 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807956 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e68d61-32a5-4ffe-987f-8a82e3181db0" containerName="payload-pre-processing" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807964 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="142d0d71-364a-4929-b3c0-5089e164982a" containerName="revoke-keys" Nov 01 14:29:22.808089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.807996 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ddd089-3c38-4334-bce3-6722f2d7a216" containerName="payload-pre-processing" Nov 01 14:29:22.811443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.811420 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:22.847949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.847920 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:29:22.858260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.858071 5473 patch_prober.go:29] interesting pod/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 01 14:29:22.858260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.858127 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:29:22.867271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.865019 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-007270-dockercfg-4tj5l\"" Nov 01 14:29:22.871130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.868621 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-007270\"" Nov 01 14:29:22.871130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.868710 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7"] Nov 01 14:29:22.903962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.903940 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:29:22.906941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.906925 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume\") pod \"d44f04d0-36f2-48f1-a806-16bad817a5da\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " Nov 01 14:29:22.906992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.906956 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46\" (UniqueName: \"kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46\") pod \"d44f04d0-36f2-48f1-a806-16bad817a5da\" (UID: \"d44f04d0-36f2-48f1-a806-16bad817a5da\") " Nov 01 14:29:22.907107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.907090 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ggt4\" (UniqueName: \"kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4\") pod \"payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:22.907156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.907127 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:22.907256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.907238 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume" pod "d44f04d0-36f2-48f1-a806-16bad817a5da" (UID: "d44f04d0-36f2-48f1-a806-16bad817a5da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:22.907382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.907367 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44f04d0-36f2-48f1-a806-16bad817a5da-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:22.909165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.909141 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46" pod "d44f04d0-36f2-48f1-a806-16bad817a5da" (UID: "d44f04d0-36f2-48f1-a806-16bad817a5da"). InnerVolumeSpecName "kube-api-access-chq46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:22.914461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914437 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv"] Nov 01 14:29:22.914850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914835 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d44f04d0-36f2-48f1-a806-16bad817a5da" containerName="payload-processing" Nov 01 14:29:22.914850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914850 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44f04d0-36f2-48f1-a806-16bad817a5da" containerName="payload-processing" Nov 01 14:29:22.914967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914880 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ef693117-223e-41d2-b4ae-15b153d2580c" containerName="payload-pre-processing" Nov 01 14:29:22.914967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914885 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef693117-223e-41d2-b4ae-15b153d2580c" containerName="payload-pre-processing" Nov 01 14:29:22.914967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914949 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" containerName="istio-proxy" Nov 01 14:29:22.914967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914956 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef693117-223e-41d2-b4ae-15b153d2580c" containerName="payload-pre-processing" Nov 01 14:29:22.914967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914963 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0533b77-9087-4fde-a8cf-f3481223bbd9" containerName="istio-proxy" Nov 01 14:29:22.915190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.914971 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44f04d0-36f2-48f1-a806-16bad817a5da" containerName="payload-processing" Nov 01 14:29:22.918125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.918108 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:22.922601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.922580 5473 generic.go:356] "Generic (PLEG): container finished" podID="d44f04d0-36f2-48f1-a806-16bad817a5da" containerID="65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84" exitCode=0 Nov 01 14:29:22.922697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.922628 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" Nov 01 14:29:22.922697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.922666 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" event={"ID":"d44f04d0-36f2-48f1-a806-16bad817a5da","Type":"ContainerDied","Data":"65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84"} Nov 01 14:29:22.922819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.922725 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr" event={"ID":"d44f04d0-36f2-48f1-a806-16bad817a5da","Type":"ContainerDied","Data":"85cfeb00c6b92a279b979e1c32baa1ac2e754e0c728fe6fcbf6ca7a61b5a8588"} Nov 01 14:29:22.922819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.922759 5473 scope.go:122] "RemoveContainer" containerID="65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84" Nov 01 14:29:22.927634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.927607 5473 generic.go:356] "Generic (PLEG): container finished" podID="ef693117-223e-41d2-b4ae-15b153d2580c" containerID="87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438" exitCode=0 Nov 01 14:29:22.928121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.927707 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" event={"ID":"ef693117-223e-41d2-b4ae-15b153d2580c","Type":"ContainerDied","Data":"87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438"} Nov 01 14:29:22.928121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.927736 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" Nov 01 14:29:22.928121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.927745 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw" event={"ID":"ef693117-223e-41d2-b4ae-15b153d2580c","Type":"ContainerDied","Data":"c5a6fcb6997a164ca509b42fc5f66e31adbc358799c6aa3cfd877f71b90b51f3"} Nov 01 14:29:22.936080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.936058 5473 scope.go:122] "RemoveContainer" containerID="65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84" Nov 01 14:29:22.936350 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:22.936330 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84\": container with ID starting with 65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84 not found: ID does not exist" containerID="65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84" Nov 01 14:29:22.936470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.936362 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84"} err="failed to get container status \"65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84\": rpc error: code = NotFound desc = could not find container \"65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84\": container with ID starting with 65b871b820a7d0a9c6b845e49e0f33ba4d8ea2cd1797661e22fe1131e7afad84 not found: ID does not exist" Nov 01 14:29:22.936470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.936384 5473 scope.go:122] "RemoveContainer" containerID="87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438" Nov 01 14:29:22.944605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.944586 5473 scope.go:122] "RemoveContainer" containerID="87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438" Nov 01 14:29:22.944878 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:22.944862 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438\": container with ID starting with 87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438 not found: ID does not exist" containerID="87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438" Nov 01 14:29:22.944929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.944886 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438"} err="failed to get container status \"87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438\": rpc error: code = NotFound desc = could not find container \"87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438\": container with ID starting with 87a5d8b717667ccd4e00a41b7266394716ebbc497593b81ff3d2c2fb9bae2438 not found: ID does not exist" Nov 01 14:29:22.972465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:22.972388 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv"] Nov 01 14:29:23.007851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.007815 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume\") pod \"ef693117-223e-41d2-b4ae-15b153d2580c\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " Nov 01 14:29:23.007851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.007856 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v\" (UniqueName: \"kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v\") pod \"ef693117-223e-41d2-b4ae-15b153d2580c\" (UID: \"ef693117-223e-41d2-b4ae-15b153d2580c\") " Nov 01 14:29:23.008100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008025 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.008172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008102 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ggt4\" (UniqueName: \"kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4\") pod \"payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:23.008249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008166 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kkt\" (UniqueName: \"kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt\") pod \"payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.008312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008250 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:23.008312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008291 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume" pod "ef693117-223e-41d2-b4ae-15b153d2580c" (UID: "ef693117-223e-41d2-b4ae-15b153d2580c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:23.008386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008353 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chq46\" (UniqueName: \"kubernetes.io/projected/d44f04d0-36f2-48f1-a806-16bad817a5da-kube-api-access-chq46\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:23.008869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.008851 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:23.010062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.010046 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v" pod "ef693117-223e-41d2-b4ae-15b153d2580c" (UID: "ef693117-223e-41d2-b4ae-15b153d2580c"). InnerVolumeSpecName "kube-api-access-jgh5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:23.065292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.065263 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ggt4\" (UniqueName: \"kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4\") pod \"payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:23.080171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.080140 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8eb5a7-15aa-44f1-8738-a34081db3f80" path="/var/lib/kubelet/pods/0f8eb5a7-15aa-44f1-8738-a34081db3f80/volumes" Nov 01 14:29:23.080679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.080660 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142d0d71-364a-4929-b3c0-5089e164982a" path="/var/lib/kubelet/pods/142d0d71-364a-4929-b3c0-5089e164982a/volumes" Nov 01 14:29:23.081075 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.081057 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b543edc5-8e5f-4a8e-9331-b66d92654b17" path="/var/lib/kubelet/pods/b543edc5-8e5f-4a8e-9331-b66d92654b17/volumes" Nov 01 14:29:23.097503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.097475 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:23.110109 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.110079 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.110257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.110209 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kkt\" (UniqueName: \"kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt\") pod \"payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.110369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.110352 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef693117-223e-41d2-b4ae-15b153d2580c-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:23.110440 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.110375 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgh5v\" (UniqueName: \"kubernetes.io/projected/ef693117-223e-41d2-b4ae-15b153d2580c-kube-api-access-jgh5v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:23.110851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.110832 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.132715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.132687 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:23.168417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.168382 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kkt\" (UniqueName: \"kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt\") pod \"payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.235027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.234959 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.390891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.390854 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:23.391207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.391188 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw"] Nov 01 14:29:23.395061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.395038 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-c0f3259b-5497fc5f5f-rjdxw"] Nov 01 14:29:23.501127 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:23.501097 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc8104c_bf33_4a46_9d04_0184d59e07c1.slice/crio-aa4f9a4f6d2ec13392c660f9d0fe480a44e2156f9ccbe991d7f37d162058847b WatchSource:0}: Error finding container aa4f9a4f6d2ec13392c660f9d0fe480a44e2156f9ccbe991d7f37d162058847b: Status 404 returned error can't find the container with id aa4f9a4f6d2ec13392c660f9d0fe480a44e2156f9ccbe991d7f37d162058847b Nov 01 14:29:23.503386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.503365 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7"] Nov 01 14:29:23.507350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.507327 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr"] Nov 01 14:29:23.512775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.512757 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-c0f3259b-6797b68c99-fnthr"] Nov 01 14:29:23.538636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.538610 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9"] Nov 01 14:29:23.543597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.543577 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.597022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.596996 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-00a787-openshift-default-dockercfg-v7d89\"" Nov 01 14:29:23.599507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.599471 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9"] Nov 01 14:29:23.603873 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:23.603847 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7dd0e6_ac53_4ebb_8b61_60dad4575c23.slice/crio-7aaeb5433a2a0c439172a1b0adcb3cc9b90fca454bca76471cdabfe7dd2c0097 WatchSource:0}: Error finding container 7aaeb5433a2a0c439172a1b0adcb3cc9b90fca454bca76471cdabfe7dd2c0097: Status 404 returned error can't find the container with id 7aaeb5433a2a0c439172a1b0adcb3cc9b90fca454bca76471cdabfe7dd2c0097 Nov 01 14:29:23.606091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.606070 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv"] Nov 01 14:29:23.615277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615252 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615415 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615456 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615544 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615591 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615662 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615704 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.615758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.615730 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59hcf\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.664606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.664580 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:23.716869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.716834 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.716886 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.716998 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717054 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717079 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717111 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717138 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717189 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717258 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59hcf\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717478 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717499 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717542 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717775 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.717927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.717816 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.719134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.719115 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.719363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.719347 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.774615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.774589 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.775791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.775775 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59hcf\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.882559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.882529 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:23.886552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.886526 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:23.952344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.947135 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" event={"ID":"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23","Type":"ContainerStarted","Data":"9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253"} Nov 01 14:29:23.952344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.947196 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" event={"ID":"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23","Type":"ContainerStarted","Data":"7aaeb5433a2a0c439172a1b0adcb3cc9b90fca454bca76471cdabfe7dd2c0097"} Nov 01 14:29:23.952344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.948070 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:23.965603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.965575 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" event={"ID":"1cc8104c-bf33-4a46-9d04-0184d59e07c1","Type":"ContainerStarted","Data":"31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5"} Nov 01 14:29:23.965748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.965636 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:23.965748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:23.965658 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" event={"ID":"1cc8104c-bf33-4a46-9d04-0184d59e07c1","Type":"ContainerStarted","Data":"aa4f9a4f6d2ec13392c660f9d0fe480a44e2156f9ccbe991d7f37d162058847b"} Nov 01 14:29:24.063924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.063829 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" podStartSLOduration=2.063815291 podStartE2EDuration="2.063815291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:24.062810973 +0000 UTC m=+703.120962813" watchObservedRunningTime="2026-11-01 14:29:24.063815291 +0000 UTC m=+703.121967137" Nov 01 14:29:24.125950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.122881 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc"] Nov 01 14:29:24.128765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.128730 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.186785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.186750 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536a87-openshift-default-dockercfg-588rv\"" Nov 01 14:29:24.187030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.186999 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc"] Nov 01 14:29:24.187475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.187437 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" podStartSLOduration=2.187424554 podStartE2EDuration="2.187424554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:24.182663421 +0000 UTC m=+703.240815270" watchObservedRunningTime="2026-11-01 14:29:24.187424554 +0000 UTC m=+703.245576430" Nov 01 14:29:24.223193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223156 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223212 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223298 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223466 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223507 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223594 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223611 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srr65\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223662 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.223755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.223740 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.242096 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.242068 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:24.289139 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:24.289108 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7142a33e_409f_4cfe_b164_2b5a919255da.slice/crio-46a98e484e5d954948963431414efd52d36f43f275d84da242b6a125dac6c93f WatchSource:0}: Error finding container 46a98e484e5d954948963431414efd52d36f43f275d84da242b6a125dac6c93f: Status 404 returned error can't find the container with id 46a98e484e5d954948963431414efd52d36f43f275d84da242b6a125dac6c93f Nov 01 14:29:24.291183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.291154 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9"] Nov 01 14:29:24.292092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.292061 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:29:24.292193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.292120 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:29:24.292193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.292150 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:29:24.324498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324443 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324500 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324639 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324612 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324726 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324757 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srr65\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324780 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324782 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324835 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324869 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.324946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.324923 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.325188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.325159 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.325290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.325265 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.325361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.325340 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.325536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.325515 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.326770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.326753 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.326823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.326794 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.382362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.382315 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.382530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.382485 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srr65\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.442322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.442290 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:24.470436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.470402 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:24.688847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.688764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:24.815105 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:24.815074 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee55d16c_72ca_4f4b_8703_7e51dad943db.slice/crio-3cf5a40637b5b4cb3d301fedf8854af2c451d89e65bdf9ea57b134310053bfb6 WatchSource:0}: Error finding container 3cf5a40637b5b4cb3d301fedf8854af2c451d89e65bdf9ea57b134310053bfb6: Status 404 returned error can't find the container with id 3cf5a40637b5b4cb3d301fedf8854af2c451d89e65bdf9ea57b134310053bfb6 Nov 01 14:29:24.816933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.816909 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc"] Nov 01 14:29:24.817824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.817794 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:29:24.817900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.817861 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:29:24.817900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.817892 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:29:24.871159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.871134 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:24.974951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.974849 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" event={"ID":"ee55d16c-72ca-4f4b-8703-7e51dad943db","Type":"ContainerStarted","Data":"3861528aa02dc423c8cb6851f69393f215b7fac6256382207f5ef8a96fc3ed04"} Nov 01 14:29:24.974951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.974908 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" event={"ID":"ee55d16c-72ca-4f4b-8703-7e51dad943db","Type":"ContainerStarted","Data":"3cf5a40637b5b4cb3d301fedf8854af2c451d89e65bdf9ea57b134310053bfb6"} Nov 01 14:29:24.976712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.976687 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" event={"ID":"7142a33e-409f-4cfe-b164-2b5a919255da","Type":"ContainerStarted","Data":"6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5"} Nov 01 14:29:24.976840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:24.976742 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" event={"ID":"7142a33e-409f-4cfe-b164-2b5a919255da","Type":"ContainerStarted","Data":"46a98e484e5d954948963431414efd52d36f43f275d84da242b6a125dac6c93f"} Nov 01 14:29:25.080924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.080884 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44f04d0-36f2-48f1-a806-16bad817a5da" path="/var/lib/kubelet/pods/d44f04d0-36f2-48f1-a806-16bad817a5da/volumes" Nov 01 14:29:25.081358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.081339 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef693117-223e-41d2-b4ae-15b153d2580c" path="/var/lib/kubelet/pods/ef693117-223e-41d2-b4ae-15b153d2580c/volumes" Nov 01 14:29:25.097983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.097938 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podStartSLOduration=1.097926535 podStartE2EDuration="1.097926535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:25.097641962 +0000 UTC m=+704.155793811" watchObservedRunningTime="2026-11-01 14:29:25.097926535 +0000 UTC m=+704.156078384" Nov 01 14:29:25.219908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.219855 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podStartSLOduration=2.219839249 podStartE2EDuration="2.219839249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:25.217492508 +0000 UTC m=+704.275644358" watchObservedRunningTime="2026-11-01 14:29:25.219839249 +0000 UTC m=+704.277991098" Nov 01 14:29:25.325398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.325353 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:25.442934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.442903 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:25.448057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.448032 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:25.567359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.567332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:25.813727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.813697 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:25.887041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.887007 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:25.892762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.892732 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:25.985558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.985531 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:25.986158 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.985579 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:25.986158 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.986031 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:29:25.986158 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:25.986133 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:29:26.097359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:26.097282 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:27.648181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.648157 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:29:27.761881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.761856 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.761890 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.761910 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.761942 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762016 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762052 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762134 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:27.762346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762129 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762264 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762297 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:27.762346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762319 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket\") pod \"7de45ce9-12db-4799-9d61-525388e0a069\" (UID: \"7de45ce9-12db-4799-9d61-525388e0a069\") " Nov 01 14:29:27.762519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762392 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:27.762552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762536 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:27.762593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762461 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:29:27.762732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762706 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.762732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762725 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.762862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762741 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7de45ce9-12db-4799-9d61-525388e0a069-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.762862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762752 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.762862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.762761 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.764179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.764146 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:29:27.764352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.764336 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:29:27.764549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.764529 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "kube-api-access-trk7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:27.764549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.764543 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token" pod "7de45ce9-12db-4799-9d61-525388e0a069" (UID: "7de45ce9-12db-4799-9d61-525388e0a069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:29:27.864389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.864358 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7de45ce9-12db-4799-9d61-525388e0a069-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.864389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.864387 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trk7g\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-kube-api-access-trk7g\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.864389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.864398 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7de45ce9-12db-4799-9d61-525388e0a069-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.864608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.864407 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7de45ce9-12db-4799-9d61-525388e0a069-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:29:27.997750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.997716 5473 generic.go:356] "Generic (PLEG): container finished" podID="7de45ce9-12db-4799-9d61-525388e0a069" containerID="54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07" exitCode=0 Nov 01 14:29:27.997911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.997786 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" Nov 01 14:29:27.997911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.997789 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" event={"ID":"7de45ce9-12db-4799-9d61-525388e0a069","Type":"ContainerDied","Data":"54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07"} Nov 01 14:29:27.997911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.997874 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv" event={"ID":"7de45ce9-12db-4799-9d61-525388e0a069","Type":"ContainerDied","Data":"d8acde460c1f8e1a02d6113233879ddbcbd79c98a0d6f8b1d28deb9ecee0228d"} Nov 01 14:29:27.997911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:27.997898 5473 scope.go:122] "RemoveContainer" containerID="54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07" Nov 01 14:29:28.008204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.008187 5473 scope.go:122] "RemoveContainer" containerID="e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2" Nov 01 14:29:28.016513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.016483 5473 scope.go:122] "RemoveContainer" containerID="54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07" Nov 01 14:29:28.016782 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:28.016759 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07\": container with ID starting with 54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07 not found: ID does not exist" containerID="54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07" Nov 01 14:29:28.016827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.016791 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07"} err="failed to get container status \"54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07\": rpc error: code = NotFound desc = could not find container \"54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07\": container with ID starting with 54e5d5b072cbb40987fa4a7015ee017814eca0f466317972cb157bd059675b07 not found: ID does not exist" Nov 01 14:29:28.016827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.016809 5473 scope.go:122] "RemoveContainer" containerID="e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2" Nov 01 14:29:28.017076 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:28.017056 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2\": container with ID starting with e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2 not found: ID does not exist" containerID="e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2" Nov 01 14:29:28.017136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.017081 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2"} err="failed to get container status \"e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2\": rpc error: code = NotFound desc = could not find container \"e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2\": container with ID starting with e4b12776934c167fb065e64e11f7269efcd749944a4b530cf2890b16a063d7d2 not found: ID does not exist" Nov 01 14:29:28.163948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.163907 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv"] Nov 01 14:29:28.168374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:28.168349 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c0f3259b-openshift-default-9577c65b5-drkrv"] Nov 01 14:29:29.079859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.079827 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de45ce9-12db-4799-9d61-525388e0a069" path="/var/lib/kubelet/pods/7de45ce9-12db-4799-9d61-525388e0a069/volumes" Nov 01 14:29:29.184932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.184896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:29.349628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.349548 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:29.550180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550148 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442"] Nov 01 14:29:29.550589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550575 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" Nov 01 14:29:29.550589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550590 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" Nov 01 14:29:29.550691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550604 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" Nov 01 14:29:29.550691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550610 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" Nov 01 14:29:29.550691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550686 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" Nov 01 14:29:29.550777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.550695 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de45ce9-12db-4799-9d61-525388e0a069" containerName="istio-proxy" Nov 01 14:29:29.555077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.555059 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.608457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.608375 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442"] Nov 01 14:29:29.653518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.653482 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5b0dcc\"" Nov 01 14:29:29.653680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.653530 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5b0dcc-dockercfg-9s7n7\"" Nov 01 14:29:29.682678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.682646 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzwz\" (UniqueName: \"kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz\") pod \"payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.682832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.682718 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.783664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.783631 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzwz\" (UniqueName: \"kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz\") pod \"payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.783820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.783698 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.784381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.784364 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.840876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.840843 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzwz\" (UniqueName: \"kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz\") pod \"payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:29.868666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:29.868595 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:30.232508 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:30.232479 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8d391e_8cd5_4057_a8c5_6570fb3b10ec.slice/crio-e38ad1ab7f16bdd84a15ae116270d43bed7da8236195a57b4f6d92d7b09b1e51 WatchSource:0}: Error finding container e38ad1ab7f16bdd84a15ae116270d43bed7da8236195a57b4f6d92d7b09b1e51: Status 404 returned error can't find the container with id e38ad1ab7f16bdd84a15ae116270d43bed7da8236195a57b4f6d92d7b09b1e51 Nov 01 14:29:30.234871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:30.234847 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442"] Nov 01 14:29:30.295259 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:30.295159 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:30.471017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:30.470980 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:30.673376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:30.673342 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:31.017291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.017256 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" event={"ID":"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec","Type":"ContainerStarted","Data":"768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d"} Nov 01 14:29:31.017487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.017312 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" event={"ID":"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec","Type":"ContainerStarted","Data":"e38ad1ab7f16bdd84a15ae116270d43bed7da8236195a57b4f6d92d7b09b1e51"} Nov 01 14:29:31.017487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.017433 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:31.106743 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:29:31.106709 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:29:31.138797 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.134569 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" podStartSLOduration=2.134550738 podStartE2EDuration="2.134550738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:31.130489201 +0000 UTC m=+710.188641048" watchObservedRunningTime="2026-11-01 14:29:31.134550738 +0000 UTC m=+710.192702588" Nov 01 14:29:31.508587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.508505 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:31.681410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.681376 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:31.877040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:31.877004 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:32.119102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:32.119067 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:32.318542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:32.318514 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:32.594129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:32.594055 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:34.980407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:34.980373 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 14:29:35.464867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:35.464837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:35.989164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:35.989138 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 14:29:36.091613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.091581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:36.282771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.282740 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:36.473720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.473685 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb"] Nov 01 14:29:36.477265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.477250 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.535528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.535313 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb"] Nov 01 14:29:36.559838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.559800 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmrp\" (UniqueName: \"kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp\") pod \"payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.560064 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.559889 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.580767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.580738 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-1f8e2c\"" Nov 01 14:29:36.580767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.580739 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-1f8e2c-dockercfg-lfdp6\"" Nov 01 14:29:36.639457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.639426 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz"] Nov 01 14:29:36.643135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.643115 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.660873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.660853 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.660979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.660944 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmrp\" (UniqueName: \"kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp\") pod \"payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.661023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.660976 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9v2p\" (UniqueName: \"kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p\") pod \"payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.661023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.661011 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.661532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.661513 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.704699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.704669 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz"] Nov 01 14:29:36.719133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.719104 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmrp\" (UniqueName: \"kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp\") pod \"payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.762078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.762044 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9v2p\" (UniqueName: \"kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p\") pod \"payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.762243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.762099 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.762851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.762835 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.790624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.790563 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:36.820076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.820046 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9v2p\" (UniqueName: \"kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p\") pod \"payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:36.955704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:36.955674 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:37.158983 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:37.158949 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18da4621_7b38_450f_ab04_2e89039b46a2.slice/crio-2eb4fa4a196bc88c331943955e8530a8f17311b7254917a87dd11ca86480bd0c WatchSource:0}: Error finding container 2eb4fa4a196bc88c331943955e8530a8f17311b7254917a87dd11ca86480bd0c: Status 404 returned error can't find the container with id 2eb4fa4a196bc88c331943955e8530a8f17311b7254917a87dd11ca86480bd0c Nov 01 14:29:37.162528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:37.162508 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb"] Nov 01 14:29:37.327407 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:37.327378 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0defb1_457c_44e2_a7c3_13c9bc08a042.slice/crio-287ee99ec1e99cc5ea812a63d08d0ac856ad97291d530644b6343a8e682bb4f1 WatchSource:0}: Error finding container 287ee99ec1e99cc5ea812a63d08d0ac856ad97291d530644b6343a8e682bb4f1: Status 404 returned error can't find the container with id 287ee99ec1e99cc5ea812a63d08d0ac856ad97291d530644b6343a8e682bb4f1 Nov 01 14:29:37.330611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:37.330590 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz"] Nov 01 14:29:38.060189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.060151 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" event={"ID":"18da4621-7b38-450f-ab04-2e89039b46a2","Type":"ContainerStarted","Data":"01130abfafcc34fb2467d26467d26395d841786c19de8491283563f5ce768259"} Nov 01 14:29:38.060447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.060230 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" event={"ID":"18da4621-7b38-450f-ab04-2e89039b46a2","Type":"ContainerStarted","Data":"2eb4fa4a196bc88c331943955e8530a8f17311b7254917a87dd11ca86480bd0c"} Nov 01 14:29:38.060447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.060260 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:38.061711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.061686 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" event={"ID":"6f0defb1-457c-44e2-a7c3-13c9bc08a042","Type":"ContainerStarted","Data":"d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c"} Nov 01 14:29:38.061817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.061729 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" event={"ID":"6f0defb1-457c-44e2-a7c3-13c9bc08a042","Type":"ContainerStarted","Data":"287ee99ec1e99cc5ea812a63d08d0ac856ad97291d530644b6343a8e682bb4f1"} Nov 01 14:29:38.061817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.061758 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:38.181878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.181822 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" podStartSLOduration=2.181805513 podStartE2EDuration="2.181805513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:38.180818025 +0000 UTC m=+717.238969875" watchObservedRunningTime="2026-11-01 14:29:38.181805513 +0000 UTC m=+717.239957364" Nov 01 14:29:38.301506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.301453 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" podStartSLOduration=2.301439932 podStartE2EDuration="2.301439932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:38.299469114 +0000 UTC m=+717.357620963" watchObservedRunningTime="2026-11-01 14:29:38.301439932 +0000 UTC m=+717.359591780" Nov 01 14:29:38.538983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.538951 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:38.761783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.761749 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:38.890779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.890706 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6"] Nov 01 14:29:38.894986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.894967 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:38.948702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.948662 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-536a87\"" Nov 01 14:29:38.950229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.950185 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6"] Nov 01 14:29:38.983823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.983770 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:38.984042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.983897 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmtv4\" (UniqueName: \"kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4\") pod \"payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:38.997752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:38.997697 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-536a87-dockercfg-2fmhq\"" Nov 01 14:29:39.051826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.051792 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p"] Nov 01 14:29:39.055596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.055574 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.084876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.084844 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.085011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.084972 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48d8\" (UniqueName: \"kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8\") pod \"payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.085074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.085015 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmtv4\" (UniqueName: \"kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4\") pod \"payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:39.085170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.085146 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:39.085765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.085743 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:39.114692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.114658 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p"] Nov 01 14:29:39.143369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.143302 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmtv4\" (UniqueName: \"kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4\") pod \"payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:39.186357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.186319 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.186816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.186404 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48d8\" (UniqueName: \"kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8\") pod \"payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.187509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.187481 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.209122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.209091 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:39.244972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.244933 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48d8\" (UniqueName: \"kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8\") pod \"payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.368776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.368747 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:39.581168 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:39.581142 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17c119a_7a1b_48d4_b093_99b268dc1569.slice/crio-d3124103c2608ca7dc2ff12101d06f93a0a6fee720f53f132e22bdcb2d272cf7 WatchSource:0}: Error finding container d3124103c2608ca7dc2ff12101d06f93a0a6fee720f53f132e22bdcb2d272cf7: Status 404 returned error can't find the container with id d3124103c2608ca7dc2ff12101d06f93a0a6fee720f53f132e22bdcb2d272cf7 Nov 01 14:29:39.583373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.583351 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6"] Nov 01 14:29:39.739169 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:39.739144 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6738a0ee_98a9_4c7d_a2d4_4d8ebc7350b7.slice/crio-e1acfc6b56b10eaa2abbed5b720a0b7b0fb45c904885510cde465576d16fd388 WatchSource:0}: Error finding container e1acfc6b56b10eaa2abbed5b720a0b7b0fb45c904885510cde465576d16fd388: Status 404 returned error can't find the container with id e1acfc6b56b10eaa2abbed5b720a0b7b0fb45c904885510cde465576d16fd388 Nov 01 14:29:39.746391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:39.742879 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p"] Nov 01 14:29:40.073902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.073867 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" event={"ID":"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7","Type":"ContainerStarted","Data":"649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d"} Nov 01 14:29:40.074085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.073910 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" event={"ID":"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7","Type":"ContainerStarted","Data":"e1acfc6b56b10eaa2abbed5b720a0b7b0fb45c904885510cde465576d16fd388"} Nov 01 14:29:40.074177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.074091 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:40.075638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.075616 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" event={"ID":"a17c119a-7a1b-48d4-b093-99b268dc1569","Type":"ContainerStarted","Data":"38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19"} Nov 01 14:29:40.075742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.075661 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" event={"ID":"a17c119a-7a1b-48d4-b093-99b268dc1569","Type":"ContainerStarted","Data":"d3124103c2608ca7dc2ff12101d06f93a0a6fee720f53f132e22bdcb2d272cf7"} Nov 01 14:29:40.075742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.075700 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:40.121181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.121130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:40.192395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.192335 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" podStartSLOduration=1.192318196 podStartE2EDuration="1.192318196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:40.190417149 +0000 UTC m=+719.248568997" watchObservedRunningTime="2026-11-01 14:29:40.192318196 +0000 UTC m=+719.250470048" Nov 01 14:29:40.301949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.301921 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:40.310983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.310939 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" podStartSLOduration=2.3109260750000002 podStartE2EDuration="2.310926075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:40.310269271 +0000 UTC m=+719.368421118" watchObservedRunningTime="2026-11-01 14:29:40.310926075 +0000 UTC m=+719.369077925" Nov 01 14:29:40.496866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.496789 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:40.973605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:40.973569 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:41.222440 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.222414 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:41.353136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.353104 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j"] Nov 01 14:29:41.358696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.358673 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.409043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.409015 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qz7\" (UniqueName: \"kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7\") pod \"payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.409241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.409169 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.417464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.417441 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j"] Nov 01 14:29:41.464587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.464558 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-00a787\"" Nov 01 14:29:41.464761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.464643 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-00a787-dockercfg-kpq4l\"" Nov 01 14:29:41.510687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.510607 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qz7\" (UniqueName: \"kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7\") pod \"payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.510833 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.510714 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.511446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.511430 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.517174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.517157 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf"] Nov 01 14:29:41.521242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.521206 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.558742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.558711 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf"] Nov 01 14:29:41.571607 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.571576 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qz7\" (UniqueName: \"kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7\") pod \"payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.611835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.611804 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7h5\" (UniqueName: \"kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5\") pod \"payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.611997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.611963 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.671450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.671424 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:41.713225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.713170 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.713393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.713270 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7h5\" (UniqueName: \"kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5\") pod \"payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.713865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.713846 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.771336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.771306 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7h5\" (UniqueName: \"kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5\") pod \"payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:41.834496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:41.834467 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:42.026188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:42.026111 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 14:29:42.041158 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:42.041131 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfb3f3b_fdcb_4bb5_9158_69c7b1d94511.slice/crio-d411c562ea56afad99acc252153db54ff60a4fbe1a6bdf6296ffcf04184ddeed WatchSource:0}: Error finding container d411c562ea56afad99acc252153db54ff60a4fbe1a6bdf6296ffcf04184ddeed: Status 404 returned error can't find the container with id d411c562ea56afad99acc252153db54ff60a4fbe1a6bdf6296ffcf04184ddeed Nov 01 14:29:42.043596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:42.043572 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j"] Nov 01 14:29:42.087729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:42.087696 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" event={"ID":"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511","Type":"ContainerStarted","Data":"d411c562ea56afad99acc252153db54ff60a4fbe1a6bdf6296ffcf04184ddeed"} Nov 01 14:29:42.201554 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:29:42.201521 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521f5798_1315_4865_ab33_a8314f7b6e15.slice/crio-ef4f457ec953cfbfcb65f021c34d2275be6d238186666873b244095803929143 WatchSource:0}: Error finding container ef4f457ec953cfbfcb65f021c34d2275be6d238186666873b244095803929143: Status 404 returned error can't find the container with id ef4f457ec953cfbfcb65f021c34d2275be6d238186666873b244095803929143 Nov 01 14:29:42.203522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:42.203502 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf"] Nov 01 14:29:43.094855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.094816 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" event={"ID":"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511","Type":"ContainerStarted","Data":"ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779"} Nov 01 14:29:43.095315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.094930 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:43.096437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.096410 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" event={"ID":"521f5798-1315-4865-ab33-a8314f7b6e15","Type":"ContainerStarted","Data":"684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d"} Nov 01 14:29:43.096529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.096455 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" event={"ID":"521f5798-1315-4865-ab33-a8314f7b6e15","Type":"ContainerStarted","Data":"ef4f457ec953cfbfcb65f021c34d2275be6d238186666873b244095803929143"} Nov 01 14:29:43.096529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.096486 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:43.213080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.213028 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" podStartSLOduration=2.21301538 podStartE2EDuration="2.21301538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:43.211728423 +0000 UTC m=+722.269880272" watchObservedRunningTime="2026-11-01 14:29:43.21301538 +0000 UTC m=+722.271167230" Nov 01 14:29:43.331871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:43.331823 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" podStartSLOduration=2.331808819 podStartE2EDuration="2.331808819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:29:43.329140727 +0000 UTC m=+722.387292578" watchObservedRunningTime="2026-11-01 14:29:43.331808819 +0000 UTC m=+722.389960667" Nov 01 14:29:45.111916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:45.111883 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:46.536007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:46.535949 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:49.071507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:49.071478 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 14:29:49.071861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:49.071541 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 14:29:49.881346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:49.881315 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:50.088471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:50.088440 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:50.270368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:50.270340 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:51.084340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:51.084310 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 14:29:51.084505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:51.084366 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 14:29:53.037010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:53.036975 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:53.606417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:53.606386 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:53.806935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:53.806904 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:54.104868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:54.104840 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 14:29:54.105337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:54.104907 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 14:29:55.060758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:55.060682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:55.254177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:55.254146 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:58.749180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:58.749145 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:29:59.823632 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:29:59.823594 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:00.172631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.172552 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f"] Nov 01 14:30:00.176847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.176823 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.179009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.178986 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh"] Nov 01 14:30:00.182586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.182569 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w"] Nov 01 14:30:00.182692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.182679 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.187472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.187452 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.222572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.222549 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst"] Nov 01 14:30:00.249259 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.248468 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.249515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.249494 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:30:00.270952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.270918 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst"] Nov 01 14:30:00.274310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.274283 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh"] Nov 01 14:30:00.297737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.297709 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld846\" (UniqueName: \"kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846\") pod \"maas-api-key-cleanup-29892390-klbbh\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.297871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.297759 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ctz\" (UniqueName: \"kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.297925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.297900 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.297996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.297980 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmjq\" (UniqueName: \"kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.298155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.298136 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892390-klbbh\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.298292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.298263 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.298416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.298362 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.298491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.298437 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65gf\" (UniqueName: \"kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.302490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.302466 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w"] Nov 01 14:30:00.304573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.304552 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f"] Nov 01 14:30:00.399632 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399600 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65gf\" (UniqueName: \"kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.399818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399643 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld846\" (UniqueName: \"kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846\") pod \"maas-api-key-cleanup-29892390-klbbh\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.399818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399673 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ctz\" (UniqueName: \"kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.399818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399691 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.399818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399714 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmjq\" (UniqueName: \"kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.399818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399769 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892390-klbbh\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.399818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399808 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.400208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.399852 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.400743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.400717 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.400846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.400717 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.400846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.400715 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892390-klbbh\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.401169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.401151 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.461616 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.461541 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld846\" (UniqueName: \"kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846\") pod \"maas-api-key-cleanup-29892390-klbbh\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.461896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.461870 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmjq\" (UniqueName: \"kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.461973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.461931 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65gf\" (UniqueName: \"kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.462459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.462442 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ctz\" (UniqueName: \"kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.492113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.492090 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:00.499665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.499649 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:00.505615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.505597 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:00.562756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.562727 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:00.893507 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:30:00.893474 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97552a7f_3e91_4faf_bef9_2be83e04e00b.slice/crio-fd37118affed358aef404c99f51575d17cc98a65918ffdf0a862dca10c8282bd WatchSource:0}: Error finding container fd37118affed358aef404c99f51575d17cc98a65918ffdf0a862dca10c8282bd: Status 404 returned error can't find the container with id fd37118affed358aef404c99f51575d17cc98a65918ffdf0a862dca10c8282bd Nov 01 14:30:00.896548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.896524 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f"] Nov 01 14:30:00.952197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.951809 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh"] Nov 01 14:30:00.953987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.953965 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w"] Nov 01 14:30:00.993157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:00.993133 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst"] Nov 01 14:30:01.052298 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:30:01.051914 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd49427_319c_4836_98c9_b844eb54b1b6.slice/crio-0d898eaf645ae7a127c21cc457d8a5677ed6c731936676e15df21e58fab4f35a WatchSource:0}: Error finding container 0d898eaf645ae7a127c21cc457d8a5677ed6c731936676e15df21e58fab4f35a: Status 404 returned error can't find the container with id 0d898eaf645ae7a127c21cc457d8a5677ed6c731936676e15df21e58fab4f35a Nov 01 14:30:01.053024 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:30:01.052788 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9d2c29_aa94_4068_b041_ef200f3793f7.slice/crio-8e133e42068c552588f9ca8bcc1f3078fa0595939c20fc71d22cd5b1ceb9f1d6 WatchSource:0}: Error finding container 8e133e42068c552588f9ca8bcc1f3078fa0595939c20fc71d22cd5b1ceb9f1d6: Status 404 returned error can't find the container with id 8e133e42068c552588f9ca8bcc1f3078fa0595939c20fc71d22cd5b1ceb9f1d6 Nov 01 14:30:01.107861 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:01.107829 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:30:01.208366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:01.208263 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" event={"ID":"fbd49427-319c-4836-98c9-b844eb54b1b6","Type":"ContainerStarted","Data":"0d898eaf645ae7a127c21cc457d8a5677ed6c731936676e15df21e58fab4f35a"} Nov 01 14:30:01.209471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:01.209446 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" event={"ID":"97552a7f-3e91-4faf-bef9-2be83e04e00b","Type":"ContainerStarted","Data":"fd37118affed358aef404c99f51575d17cc98a65918ffdf0a862dca10c8282bd"} Nov 01 14:30:01.210634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:01.210611 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" event={"ID":"30669cfe-c507-4635-b446-b0b86ce13693","Type":"ContainerStarted","Data":"2bb39524e9c7bb9f7907451edb21a2c3710f8fbc1871f9174b71903f802945d6"} Nov 01 14:30:01.211938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:01.211919 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" event={"ID":"ed9d2c29-aa94-4068-b041-ef200f3793f7","Type":"ContainerStarted","Data":"8e133e42068c552588f9ca8bcc1f3078fa0595939c20fc71d22cd5b1ceb9f1d6"} Nov 01 14:30:02.219325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.219284 5473 generic.go:356] "Generic (PLEG): container finished" podID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerID="202b0434a714788ab78731ba4a16643e9a8950363a850c70a2ec158bf5c2118b" exitCode=60 Nov 01 14:30:02.219764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.219373 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" event={"ID":"fbd49427-319c-4836-98c9-b844eb54b1b6","Type":"ContainerDied","Data":"202b0434a714788ab78731ba4a16643e9a8950363a850c70a2ec158bf5c2118b"} Nov 01 14:30:02.219764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.219623 5473 scope.go:122] "RemoveContainer" containerID="202b0434a714788ab78731ba4a16643e9a8950363a850c70a2ec158bf5c2118b" Nov 01 14:30:02.221552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.221532 5473 generic.go:356] "Generic (PLEG): container finished" podID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerID="6162457069f291cc69eb8107ac0f1851a1b9be493802f84496295219a66e5311" exitCode=60 Nov 01 14:30:02.221650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.221612 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" event={"ID":"97552a7f-3e91-4faf-bef9-2be83e04e00b","Type":"ContainerDied","Data":"6162457069f291cc69eb8107ac0f1851a1b9be493802f84496295219a66e5311"} Nov 01 14:30:02.221820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.221801 5473 scope.go:122] "RemoveContainer" containerID="6162457069f291cc69eb8107ac0f1851a1b9be493802f84496295219a66e5311" Nov 01 14:30:02.223492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.223438 5473 generic.go:356] "Generic (PLEG): container finished" podID="30669cfe-c507-4635-b446-b0b86ce13693" containerID="cf9a836fc8cec29bb486ab0f796d2b9c05599708e697a71346feaf1d8c172b1e" exitCode=60 Nov 01 14:30:02.223656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.223520 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" event={"ID":"30669cfe-c507-4635-b446-b0b86ce13693","Type":"ContainerDied","Data":"cf9a836fc8cec29bb486ab0f796d2b9c05599708e697a71346feaf1d8c172b1e"} Nov 01 14:30:02.223712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.223696 5473 scope.go:122] "RemoveContainer" containerID="cf9a836fc8cec29bb486ab0f796d2b9c05599708e697a71346feaf1d8c172b1e" Nov 01 14:30:02.225120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.225098 5473 generic.go:356] "Generic (PLEG): container finished" podID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerID="cdb3c5d4e63c57564f470db6cf917bf6a237d38abdac37bd50b81f4e5d4e6b3f" exitCode=7 Nov 01 14:30:02.225258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.225152 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" event={"ID":"ed9d2c29-aa94-4068-b041-ef200f3793f7","Type":"ContainerDied","Data":"cdb3c5d4e63c57564f470db6cf917bf6a237d38abdac37bd50b81f4e5d4e6b3f"} Nov 01 14:30:02.225369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:02.225355 5473 scope.go:122] "RemoveContainer" containerID="cdb3c5d4e63c57564f470db6cf917bf6a237d38abdac37bd50b81f4e5d4e6b3f" Nov 01 14:30:03.234977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.234941 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" event={"ID":"30669cfe-c507-4635-b446-b0b86ce13693","Type":"ContainerStarted","Data":"81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2"} Nov 01 14:30:03.235432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.235261 5473 scope.go:122] "RemoveContainer" containerID="81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2" Nov 01 14:30:03.235578 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:03.235554 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w_odh-ai-gateway-infra(30669cfe-c507-4635-b446-b0b86ce13693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" podUID="30669cfe-c507-4635-b446-b0b86ce13693" Nov 01 14:30:03.237282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.237261 5473 generic.go:356] "Generic (PLEG): container finished" podID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerID="18ba0010d305cad90c1aa19a83c436ce7f6844cca6b6676c2c36b8478f73ad01" exitCode=7 Nov 01 14:30:03.237394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.237346 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" event={"ID":"ed9d2c29-aa94-4068-b041-ef200f3793f7","Type":"ContainerDied","Data":"18ba0010d305cad90c1aa19a83c436ce7f6844cca6b6676c2c36b8478f73ad01"} Nov 01 14:30:03.237455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.237407 5473 scope.go:122] "RemoveContainer" containerID="cdb3c5d4e63c57564f470db6cf917bf6a237d38abdac37bd50b81f4e5d4e6b3f" Nov 01 14:30:03.237664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.237646 5473 scope.go:122] "RemoveContainer" containerID="18ba0010d305cad90c1aa19a83c436ce7f6844cca6b6676c2c36b8478f73ad01" Nov 01 14:30:03.237938 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:03.237913 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst_odh-ai-gateway-infra(ed9d2c29-aa94-4068-b041-ef200f3793f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" Nov 01 14:30:03.239328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.239307 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" event={"ID":"fbd49427-319c-4836-98c9-b844eb54b1b6","Type":"ContainerStarted","Data":"18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c"} Nov 01 14:30:03.241384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.241365 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" event={"ID":"97552a7f-3e91-4faf-bef9-2be83e04e00b","Type":"ContainerStarted","Data":"feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785"} Nov 01 14:30:03.241550 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.241537 5473 scope.go:122] "RemoveContainer" containerID="feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785" Nov 01 14:30:03.241730 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:03.241711 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f_odh-ai-gateway-infra(97552a7f-3e91-4faf-bef9-2be83e04e00b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" Nov 01 14:30:03.701823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:03.701710 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" podStartSLOduration=3.701691243 podStartE2EDuration="3.701691243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:03.69983446 +0000 UTC m=+742.757986324" watchObservedRunningTime="2026-11-01 14:30:03.701691243 +0000 UTC m=+742.759843092" Nov 01 14:30:04.249688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.249655 5473 generic.go:356] "Generic (PLEG): container finished" podID="30669cfe-c507-4635-b446-b0b86ce13693" containerID="81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2" exitCode=60 Nov 01 14:30:04.250177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.249734 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" event={"ID":"30669cfe-c507-4635-b446-b0b86ce13693","Type":"ContainerDied","Data":"81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2"} Nov 01 14:30:04.250177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.249812 5473 scope.go:122] "RemoveContainer" containerID="cf9a836fc8cec29bb486ab0f796d2b9c05599708e697a71346feaf1d8c172b1e" Nov 01 14:30:04.250177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.250039 5473 scope.go:122] "RemoveContainer" containerID="81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2" Nov 01 14:30:04.250388 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:04.250335 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w_odh-ai-gateway-infra(30669cfe-c507-4635-b446-b0b86ce13693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" podUID="30669cfe-c507-4635-b446-b0b86ce13693" Nov 01 14:30:04.251627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.251607 5473 scope.go:122] "RemoveContainer" containerID="18ba0010d305cad90c1aa19a83c436ce7f6844cca6b6676c2c36b8478f73ad01" Nov 01 14:30:04.251843 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:04.251813 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst_odh-ai-gateway-infra(ed9d2c29-aa94-4068-b041-ef200f3793f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" Nov 01 14:30:04.253271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.253250 5473 generic.go:356] "Generic (PLEG): container finished" podID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerID="18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c" exitCode=60 Nov 01 14:30:04.253363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.253307 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" event={"ID":"fbd49427-319c-4836-98c9-b844eb54b1b6","Type":"ContainerDied","Data":"18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c"} Nov 01 14:30:04.253527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.253504 5473 scope.go:122] "RemoveContainer" containerID="18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c" Nov 01 14:30:04.253719 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:04.253702 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892390-klbbh_odh-ai-gateway-infra(fbd49427-319c-4836-98c9-b844eb54b1b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" Nov 01 14:30:04.255137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.255115 5473 generic.go:356] "Generic (PLEG): container finished" podID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerID="feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785" exitCode=60 Nov 01 14:30:04.255234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.255186 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" event={"ID":"97552a7f-3e91-4faf-bef9-2be83e04e00b","Type":"ContainerDied","Data":"feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785"} Nov 01 14:30:04.255366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.255354 5473 scope.go:122] "RemoveContainer" containerID="feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785" Nov 01 14:30:04.255534 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:04.255518 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f_odh-ai-gateway-infra(97552a7f-3e91-4faf-bef9-2be83e04e00b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" Nov 01 14:30:04.265613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.265589 5473 scope.go:122] "RemoveContainer" containerID="202b0434a714788ab78731ba4a16643e9a8950363a850c70a2ec158bf5c2118b" Nov 01 14:30:04.277590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.277573 5473 scope.go:122] "RemoveContainer" containerID="6162457069f291cc69eb8107ac0f1851a1b9be493802f84496295219a66e5311" Nov 01 14:30:04.388482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:04.388449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:05.262983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.262954 5473 scope.go:122] "RemoveContainer" containerID="81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2" Nov 01 14:30:05.263697 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:05.263242 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w_odh-ai-gateway-infra(30669cfe-c507-4635-b446-b0b86ce13693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" podUID="30669cfe-c507-4635-b446-b0b86ce13693" Nov 01 14:30:05.264535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.264512 5473 scope.go:122] "RemoveContainer" containerID="18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c" Nov 01 14:30:05.264764 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:05.264745 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892390-klbbh_odh-ai-gateway-infra(fbd49427-319c-4836-98c9-b844eb54b1b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" Nov 01 14:30:05.265741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.265720 5473 generic.go:356] "Generic (PLEG): container finished" podID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerID="3861528aa02dc423c8cb6851f69393f215b7fac6256382207f5ef8a96fc3ed04" exitCode=0 Nov 01 14:30:05.265825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.265788 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" event={"ID":"ee55d16c-72ca-4f4b-8703-7e51dad943db","Type":"ContainerDied","Data":"3861528aa02dc423c8cb6851f69393f215b7fac6256382207f5ef8a96fc3ed04"} Nov 01 14:30:05.266126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.266109 5473 scope.go:122] "RemoveContainer" containerID="3861528aa02dc423c8cb6851f69393f215b7fac6256382207f5ef8a96fc3ed04" Nov 01 14:30:05.267347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.267321 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:30:05.267430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.267384 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:30:05.267430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.267413 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:30:05.267519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:05.267505 5473 scope.go:122] "RemoveContainer" containerID="feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785" Nov 01 14:30:05.267703 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:05.267685 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f_odh-ai-gateway-infra(97552a7f-3e91-4faf-bef9-2be83e04e00b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" Nov 01 14:30:06.281847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:06.281811 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" event={"ID":"ee55d16c-72ca-4f4b-8703-7e51dad943db","Type":"ContainerStarted","Data":"3507a8a791e01960554197d8aa9f7940b8ddc3a9f7ec3e92c058d78666a28bb0"} Nov 01 14:30:06.448985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:06.448547 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:30:06.448985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:06.448682 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 14:30:06.448985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:06.448728 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 14:30:07.406418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:07.406383 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:07.448397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:07.448353 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 14:30:07.448698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:07.448648 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 14:30:08.007658 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:08.007623 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:08.238120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:08.238086 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:08.456240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:08.455614 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 14:30:08.456240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:08.455680 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 14:30:09.446240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:09.445631 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 14:30:09.446240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:09.445692 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 14:30:09.584241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:09.572060 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:10.444241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:10.443659 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Nov 01 14:30:10.444241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:10.443724 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Nov 01 14:30:10.985441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:10.985355 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:30:11.447375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:11.447344 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:30:11.448480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:11.448456 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:30:12.028375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:12.028336 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:30:12.028847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:12.028388 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:30:14.074500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.074474 5473 scope.go:122] "RemoveContainer" containerID="18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c" Nov 01 14:30:14.108106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.108071 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:14.336351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.336324 5473 generic.go:356] "Generic (PLEG): container finished" podID="7142a33e-409f-4cfe-b164-2b5a919255da" containerID="6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5" exitCode=0 Nov 01 14:30:14.336480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.336380 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" event={"ID":"7142a33e-409f-4cfe-b164-2b5a919255da","Type":"ContainerDied","Data":"6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5"} Nov 01 14:30:14.336729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.336715 5473 scope.go:122] "RemoveContainer" containerID="6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5" Nov 01 14:30:14.338051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.338020 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:30:14.338172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.338089 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:30:14.338172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:14.338122 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:30:15.075508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.075477 5473 scope.go:122] "RemoveContainer" containerID="feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785" Nov 01 14:30:15.076113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.075512 5473 scope.go:122] "RemoveContainer" containerID="81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2" Nov 01 14:30:15.357905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.357875 5473 generic.go:356] "Generic (PLEG): container finished" podID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerID="ce6eb97d31f8ee5ec05e51733b63f4d976b083fb21844cd0334ab6ebd27ebe45" exitCode=60 Nov 01 14:30:15.358005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.357937 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" event={"ID":"fbd49427-319c-4836-98c9-b844eb54b1b6","Type":"ContainerDied","Data":"ce6eb97d31f8ee5ec05e51733b63f4d976b083fb21844cd0334ab6ebd27ebe45"} Nov 01 14:30:15.358005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.357989 5473 scope.go:122] "RemoveContainer" containerID="18d1b052b668f6c11ed58e721b643c1db9f78dcac0c0439ac311b14a2c07482c" Nov 01 14:30:15.358467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.358447 5473 scope.go:122] "RemoveContainer" containerID="ce6eb97d31f8ee5ec05e51733b63f4d976b083fb21844cd0334ab6ebd27ebe45" Nov 01 14:30:15.358768 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:15.358741 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892390-klbbh_odh-ai-gateway-infra(fbd49427-319c-4836-98c9-b844eb54b1b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" Nov 01 14:30:15.360339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.360318 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" event={"ID":"7142a33e-409f-4cfe-b164-2b5a919255da","Type":"ContainerStarted","Data":"e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2"} Nov 01 14:30:15.887738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.887705 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:30:15.887931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.887901 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 14:30:15.888026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:15.887959 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 14:30:16.370246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.370201 5473 generic.go:356] "Generic (PLEG): container finished" podID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerID="5444f93b3ad567d2927f80114fdd630be55cf46f6e91841282752a8bc749690a" exitCode=60 Nov 01 14:30:16.370662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.370275 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" event={"ID":"97552a7f-3e91-4faf-bef9-2be83e04e00b","Type":"ContainerDied","Data":"5444f93b3ad567d2927f80114fdd630be55cf46f6e91841282752a8bc749690a"} Nov 01 14:30:16.370662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.370329 5473 scope.go:122] "RemoveContainer" containerID="feb1dad9634dbc186af157fa90403bf2c9b6bd5ef44e8b1056d7624262b73785" Nov 01 14:30:16.370791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.370668 5473 scope.go:122] "RemoveContainer" containerID="5444f93b3ad567d2927f80114fdd630be55cf46f6e91841282752a8bc749690a" Nov 01 14:30:16.370951 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:16.370932 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f_odh-ai-gateway-infra(97552a7f-3e91-4faf-bef9-2be83e04e00b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" Nov 01 14:30:16.372182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.372163 5473 generic.go:356] "Generic (PLEG): container finished" podID="30669cfe-c507-4635-b446-b0b86ce13693" containerID="a523304c03a3a359332f9b6f9370af8fb521603a0adcbf039cb670ad41453ae7" exitCode=60 Nov 01 14:30:16.372297 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.372239 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" event={"ID":"30669cfe-c507-4635-b446-b0b86ce13693","Type":"ContainerDied","Data":"a523304c03a3a359332f9b6f9370af8fb521603a0adcbf039cb670ad41453ae7"} Nov 01 14:30:16.372663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.372650 5473 scope.go:122] "RemoveContainer" containerID="a523304c03a3a359332f9b6f9370af8fb521603a0adcbf039cb670ad41453ae7" Nov 01 14:30:16.372860 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:16.372842 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w_odh-ai-gateway-infra(30669cfe-c507-4635-b446-b0b86ce13693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" podUID="30669cfe-c507-4635-b446-b0b86ce13693" Nov 01 14:30:16.382206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.382183 5473 scope.go:122] "RemoveContainer" containerID="81fc5930a79811dc7a3765b3a582f405738ab779ed2e59b554d612ea876abcb2" Nov 01 14:30:16.498121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.498089 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh"] Nov 01 14:30:16.674613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.674588 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:16.809999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.809968 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca\") pod \"fbd49427-319c-4836-98c9-b844eb54b1b6\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " Nov 01 14:30:16.810190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.810017 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846\" (UniqueName: \"kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846\") pod \"fbd49427-319c-4836-98c9-b844eb54b1b6\" (UID: \"fbd49427-319c-4836-98c9-b844eb54b1b6\") " Nov 01 14:30:16.810404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.810383 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca" pod "fbd49427-319c-4836-98c9-b844eb54b1b6" (UID: "fbd49427-319c-4836-98c9-b844eb54b1b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:30:16.812136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.812104 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846" pod "fbd49427-319c-4836-98c9-b844eb54b1b6" (UID: "fbd49427-319c-4836-98c9-b844eb54b1b6"). InnerVolumeSpecName "kube-api-access-ld846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:30:16.886906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.886868 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 14:30:16.887071 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.886919 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 14:30:16.911836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.911763 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbd49427-319c-4836-98c9-b844eb54b1b6-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:16.911836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:16.911789 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld846\" (UniqueName: \"kubernetes.io/projected/fbd49427-319c-4836-98c9-b844eb54b1b6-kube-api-access-ld846\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:17.074573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.074543 5473 scope.go:122] "RemoveContainer" containerID="18ba0010d305cad90c1aa19a83c436ce7f6844cca6b6676c2c36b8478f73ad01" Nov 01 14:30:17.387175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.387135 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" event={"ID":"ed9d2c29-aa94-4068-b041-ef200f3793f7","Type":"ContainerStarted","Data":"937d7640f48ddfe3d501aa2c9d4db9ac9e6eda6b2f82e6a01f2369ba1ee1580a"} Nov 01 14:30:17.388455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.388431 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" Nov 01 14:30:17.388586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.388450 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh" event={"ID":"fbd49427-319c-4836-98c9-b844eb54b1b6","Type":"ContainerDied","Data":"0d898eaf645ae7a127c21cc457d8a5677ed6c731936676e15df21e58fab4f35a"} Nov 01 14:30:17.388586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.388501 5473 scope.go:122] "RemoveContainer" containerID="ce6eb97d31f8ee5ec05e51733b63f4d976b083fb21844cd0334ab6ebd27ebe45" Nov 01 14:30:17.495177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.495146 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f"] Nov 01 14:30:17.507061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.507013 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" podStartSLOduration=17.506999773 podStartE2EDuration="17.506999773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:17.505779728 +0000 UTC m=+756.563931600" watchObservedRunningTime="2026-11-01 14:30:17.506999773 +0000 UTC m=+756.565151621" Nov 01 14:30:17.658689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.658368 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w"] Nov 01 14:30:17.684007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.683967 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:17.758630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.758563 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh"] Nov 01 14:30:17.763741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.763698 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892390-klbbh"] Nov 01 14:30:17.817124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.817104 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:17.821805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.821787 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf\" (UniqueName: \"kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf\") pod \"97552a7f-3e91-4faf-bef9-2be83e04e00b\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " Nov 01 14:30:17.821865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.821822 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca\") pod \"97552a7f-3e91-4faf-bef9-2be83e04e00b\" (UID: \"97552a7f-3e91-4faf-bef9-2be83e04e00b\") " Nov 01 14:30:17.822116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.822096 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca" pod "97552a7f-3e91-4faf-bef9-2be83e04e00b" (UID: "97552a7f-3e91-4faf-bef9-2be83e04e00b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:30:17.822304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.822290 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97552a7f-3e91-4faf-bef9-2be83e04e00b-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:17.823722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.823696 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf" pod "97552a7f-3e91-4faf-bef9-2be83e04e00b" (UID: "97552a7f-3e91-4faf-bef9-2be83e04e00b"). InnerVolumeSpecName "kube-api-access-z65gf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:30:17.888047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.887602 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 14:30:17.888047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.887664 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 14:30:17.922760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.922726 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca\") pod \"30669cfe-c507-4635-b446-b0b86ce13693\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " Nov 01 14:30:17.922928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.922825 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz\" (UniqueName: \"kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz\") pod \"30669cfe-c507-4635-b446-b0b86ce13693\" (UID: \"30669cfe-c507-4635-b446-b0b86ce13693\") " Nov 01 14:30:17.923232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.923189 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65gf\" (UniqueName: \"kubernetes.io/projected/97552a7f-3e91-4faf-bef9-2be83e04e00b-kube-api-access-z65gf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:17.923350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.923209 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca" pod "30669cfe-c507-4635-b446-b0b86ce13693" (UID: "30669cfe-c507-4635-b446-b0b86ce13693"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:30:17.949757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:17.949682 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz" pod "30669cfe-c507-4635-b446-b0b86ce13693" (UID: "30669cfe-c507-4635-b446-b0b86ce13693"). InnerVolumeSpecName "kube-api-access-b7ctz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:30:18.023739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.023712 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30669cfe-c507-4635-b446-b0b86ce13693-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:18.023739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.023736 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ctz\" (UniqueName: \"kubernetes.io/projected/30669cfe-c507-4635-b446-b0b86ce13693-kube-api-access-b7ctz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:18.396962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.396872 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" Nov 01 14:30:18.396962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.396874 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w" event={"ID":"30669cfe-c507-4635-b446-b0b86ce13693","Type":"ContainerDied","Data":"2bb39524e9c7bb9f7907451edb21a2c3710f8fbc1871f9174b71903f802945d6"} Nov 01 14:30:18.397471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.396997 5473 scope.go:122] "RemoveContainer" containerID="a523304c03a3a359332f9b6f9370af8fb521603a0adcbf039cb670ad41453ae7" Nov 01 14:30:18.398958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.398934 5473 generic.go:356] "Generic (PLEG): container finished" podID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerID="937d7640f48ddfe3d501aa2c9d4db9ac9e6eda6b2f82e6a01f2369ba1ee1580a" exitCode=60 Nov 01 14:30:18.399048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.399004 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" event={"ID":"ed9d2c29-aa94-4068-b041-ef200f3793f7","Type":"ContainerDied","Data":"937d7640f48ddfe3d501aa2c9d4db9ac9e6eda6b2f82e6a01f2369ba1ee1580a"} Nov 01 14:30:18.399415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.399397 5473 scope.go:122] "RemoveContainer" containerID="937d7640f48ddfe3d501aa2c9d4db9ac9e6eda6b2f82e6a01f2369ba1ee1580a" Nov 01 14:30:18.399678 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:18.399655 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst_odh-ai-gateway-infra(ed9d2c29-aa94-4068-b041-ef200f3793f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" Nov 01 14:30:18.401353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.401338 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" Nov 01 14:30:18.401353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.401347 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f" event={"ID":"97552a7f-3e91-4faf-bef9-2be83e04e00b","Type":"ContainerDied","Data":"fd37118affed358aef404c99f51575d17cc98a65918ffdf0a862dca10c8282bd"} Nov 01 14:30:18.407739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.407723 5473 scope.go:122] "RemoveContainer" containerID="18ba0010d305cad90c1aa19a83c436ce7f6844cca6b6676c2c36b8478f73ad01" Nov 01 14:30:18.420285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.420271 5473 scope.go:122] "RemoveContainer" containerID="5444f93b3ad567d2927f80114fdd630be55cf46f6e91841282752a8bc749690a" Nov 01 14:30:18.529460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.525029 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst"] Nov 01 14:30:18.675681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.675647 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w"] Nov 01 14:30:18.681539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.681518 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892390-9rs7w"] Nov 01 14:30:18.808146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.808110 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f"] Nov 01 14:30:18.811228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.811198 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892390-76p6f"] Nov 01 14:30:18.844239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.843836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:18.887766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.887735 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 14:30:18.887909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:18.887781 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 14:30:19.081131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.081101 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30669cfe-c507-4635-b446-b0b86ce13693" path="/var/lib/kubelet/pods/30669cfe-c507-4635-b446-b0b86ce13693/volumes" Nov 01 14:30:19.081539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.081525 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" path="/var/lib/kubelet/pods/97552a7f-3e91-4faf-bef9-2be83e04e00b/volumes" Nov 01 14:30:19.081914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.081902 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" path="/var/lib/kubelet/pods/fbd49427-319c-4836-98c9-b844eb54b1b6/volumes" Nov 01 14:30:19.587902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.587756 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:19.699660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.699550 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:19.764318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.764283 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca\") pod \"ed9d2c29-aa94-4068-b041-ef200f3793f7\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " Nov 01 14:30:19.764492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.764333 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq\" (UniqueName: \"kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq\") pod \"ed9d2c29-aa94-4068-b041-ef200f3793f7\" (UID: \"ed9d2c29-aa94-4068-b041-ef200f3793f7\") " Nov 01 14:30:19.764773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.764746 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca" pod "ed9d2c29-aa94-4068-b041-ef200f3793f7" (UID: "ed9d2c29-aa94-4068-b041-ef200f3793f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:30:19.766443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.766423 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq" pod "ed9d2c29-aa94-4068-b041-ef200f3793f7" (UID: "ed9d2c29-aa94-4068-b041-ef200f3793f7"). InnerVolumeSpecName "kube-api-access-vxmjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:30:19.865457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.865427 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9d2c29-aa94-4068-b041-ef200f3793f7-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:19.865457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.865456 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmjq\" (UniqueName: \"kubernetes.io/projected/ed9d2c29-aa94-4068-b041-ef200f3793f7-kube-api-access-vxmjq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:30:19.887438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.887406 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 01 14:30:19.887582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:19.887462 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 01 14:30:20.274033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.273991 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:20.422846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.422814 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" Nov 01 14:30:20.423010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.422852 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst" event={"ID":"ed9d2c29-aa94-4068-b041-ef200f3793f7","Type":"ContainerDied","Data":"8e133e42068c552588f9ca8bcc1f3078fa0595939c20fc71d22cd5b1ceb9f1d6"} Nov 01 14:30:20.423010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.422918 5473 scope.go:122] "RemoveContainer" containerID="937d7640f48ddfe3d501aa2c9d4db9ac9e6eda6b2f82e6a01f2369ba1ee1580a" Nov 01 14:30:20.455022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.454979 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:20.591973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.591889 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst"] Nov 01 14:30:20.595441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.595421 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892390-zrvst"] Nov 01 14:30:20.890645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.890567 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:30:20.890994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.890960 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:30:20.891721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:20.891700 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:30:21.082402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:21.082372 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" path="/var/lib/kubelet/pods/ed9d2c29-aa94-4068-b041-ef200f3793f7/volumes" Nov 01 14:30:23.321136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:23.321104 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:25.700430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700393 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh"] Nov 01 14:30:25.700878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700864 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.700921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700880 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.700921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700892 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.700921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700899 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.700921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700917 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700923 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700930 5473 cpu_manager.go:414] "containerMap: removing container" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700934 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700943 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700947 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700954 5473 cpu_manager.go:414] "containerMap: removing container" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700959 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700972 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700978 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700988 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.700996 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701003 5473 cpu_manager.go:414] "containerMap: removing container" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701007 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701018 5473 cpu_manager.go:414] "containerMap: removing container" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701023 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701031 5473 cpu_manager.go:414] "containerMap: removing container" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701036 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701105 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701115 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701121 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701128 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701135 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701144 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701154 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701161 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd49427-319c-4836-98c9-b844eb54b1b6" containerName="cleanup" Nov 01 14:30:25.701556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.701167 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:25.704380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.704365 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.730786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.730760 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.730939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.730798 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.730939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.730884 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6ht\" (UniqueName: \"kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.731052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.730963 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.731052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.731010 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.731134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.731115 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.807765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.807731 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"openshift-service-ca.crt\"" Nov 01 14:30:25.807765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.807731 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"kube-root-ca.crt\"" Nov 01 14:30:25.808009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.807794 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"default-dockercfg-nqggd\"" Nov 01 14:30:25.808009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.807820 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5b0dcc\"/\"simulator-w3-5b0dcc-kserve-self-signed-certs\"" Nov 01 14:30:25.810261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.810236 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh"] Nov 01 14:30:25.814179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.814157 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:25.831588 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.831568 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.831705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.831599 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.831705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.831635 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6ht\" (UniqueName: \"kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.831705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.831662 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.831705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.831693 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.831925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.831761 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.832151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.832120 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.832255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.832152 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.832255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.832170 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.834025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.834001 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.834510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.834489 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:25.888435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:25.888403 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6ht\" (UniqueName: \"kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht\") pod \"simulator-w3-5b0dcc-kserve-5797986977-gtbbh\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:26.015718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:26.015688 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:26.380609 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:30:26.380572 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf935c7db_e3c2_49f2_9329_96b7b48b02d3.slice/crio-c2749370441bfb8cb07b6eb9bbf8a7591e54c62bfbacb45d8c29dae686381da3 WatchSource:0}: Error finding container c2749370441bfb8cb07b6eb9bbf8a7591e54c62bfbacb45d8c29dae686381da3: Status 404 returned error can't find the container with id c2749370441bfb8cb07b6eb9bbf8a7591e54c62bfbacb45d8c29dae686381da3 Nov 01 14:30:26.383182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:26.383161 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh"] Nov 01 14:30:26.462409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:26.462378 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" event={"ID":"f935c7db-e3c2-49f2-9329-96b7b48b02d3","Type":"ContainerStarted","Data":"e66503467f0a237b0f7cb1833e3793aecb28c9c5613de2b90f01a7fc1b5c61a2"} Nov 01 14:30:26.462559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:26.462423 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" event={"ID":"f935c7db-e3c2-49f2-9329-96b7b48b02d3","Type":"ContainerStarted","Data":"c2749370441bfb8cb07b6eb9bbf8a7591e54c62bfbacb45d8c29dae686381da3"} Nov 01 14:30:26.578117 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:26.578066 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podStartSLOduration=1.578053782 podStartE2EDuration="1.578053782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:26.576508166 +0000 UTC m=+765.634660017" watchObservedRunningTime="2026-11-01 14:30:26.578053782 +0000 UTC m=+765.636205630" Nov 01 14:30:27.016475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:27.016445 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:28.034903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:28.034869 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:28.930782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:28.930749 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:29.098628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:29.098594 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:29.497859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:29.497824 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:29.685061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:29.685018 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:31.107589 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:30:31.107555 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:30:32.709918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:32.709887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:35.279655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:35.279622 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:36.016679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:36.016643 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:36.028991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:36.028963 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:36.531887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:36.531857 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:30:37.189901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:37.189860 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:41.603275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:41.603242 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:46.007553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:46.007519 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:46.826018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:46.825986 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:50.407035 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:50.406994 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:51.897384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.897353 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm"] Nov 01 14:30:51.897838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.897809 5473 cpu_manager.go:414] "containerMap: removing container" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:51.897838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.897820 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:51.897909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.897878 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="97552a7f-3e91-4faf-bef9-2be83e04e00b" containerName="cleanup" Nov 01 14:30:51.897909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.897889 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9d2c29-aa94-4068-b041-ef200f3793f7" containerName="cleanup" Nov 01 14:30:51.897909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.897897 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="30669cfe-c507-4635-b446-b0b86ce13693" containerName="cleanup" Nov 01 14:30:51.902321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.902303 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:51.954001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.953975 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm"] Nov 01 14:30:51.983697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.983663 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:51.983697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.983700 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:51.983876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.983801 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:51.983876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.983825 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:51.983981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.983934 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:51.984039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:51.984016 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jg8\" (UniqueName: \"kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.001504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.001481 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"openshift-service-ca.crt\"" Nov 01 14:30:52.001626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.001514 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"default-dockercfg-9t7qz\"" Nov 01 14:30:52.001626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.001521 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"premium-w2-007270-kserve-self-signed-certs\"" Nov 01 14:30:52.004386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.004372 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-007270\"/\"kube-root-ca.crt\"" Nov 01 14:30:52.016291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.016267 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:52.085418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085386 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085423 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085475 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085531 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jg8\" (UniqueName: \"kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085573 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085599 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085832 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085874 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.085912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.085899 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.087787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.087762 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.088060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.088042 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.144517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.144486 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jg8\" (UniqueName: \"kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8\") pod \"premium-w2-007270-kserve-666c6ff7d-rr6dm\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.214666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.214582 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:52.596479 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:30:52.596452 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e52a4d1_8264_4b3f_980e_971cf6ca16cd.slice/crio-6fc052490a8f83378a366a5d5a3994652642705852c1c6b7d171964f24117138 WatchSource:0}: Error finding container 6fc052490a8f83378a366a5d5a3994652642705852c1c6b7d171964f24117138: Status 404 returned error can't find the container with id 6fc052490a8f83378a366a5d5a3994652642705852c1c6b7d171964f24117138 Nov 01 14:30:52.605245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.599168 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm"] Nov 01 14:30:52.614060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:52.614024 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" event={"ID":"4e52a4d1-8264-4b3f-980e-971cf6ca16cd","Type":"ContainerStarted","Data":"6fc052490a8f83378a366a5d5a3994652642705852c1c6b7d171964f24117138"} Nov 01 14:30:53.621744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:53.621706 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" event={"ID":"4e52a4d1-8264-4b3f-980e-971cf6ca16cd","Type":"ContainerStarted","Data":"a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a"} Nov 01 14:30:53.737307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:53.737259 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podStartSLOduration=2.737245822 podStartE2EDuration="2.737245822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:30:53.736181801 +0000 UTC m=+792.794333652" watchObservedRunningTime="2026-11-01 14:30:53.737245822 +0000 UTC m=+792.795397668" Nov 01 14:30:54.214882 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:54.214853 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:30:54.808161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:54.808129 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:56.981013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:56.980982 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:59.187689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:59.187653 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:30:59.919146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:30:59.919118 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:01.107197 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:31:01.107161 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:31:02.215697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:02.215653 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:31:02.228247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:02.228209 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:31:02.690916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:02.690885 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:31:03.622403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:03.622335 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:08.000395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:08.000358 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:12.495666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:12.495635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:15.689276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:15.689207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:16.903614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:16.903585 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:21.243581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:21.243505 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:25.802745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:25.802698 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:30.151359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:30.151329 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:31.107237 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:31:31.107178 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:31:34.595956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:34.595921 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:39.007232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:39.007187 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:43.264653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.264620 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7"] Nov 01 14:31:43.268648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.268630 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.321055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.321025 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7"] Nov 01 14:31:43.368444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.368403 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"openshift-service-ca.crt\"" Nov 01 14:31:43.368444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.368439 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"kube-root-ca.crt\"" Nov 01 14:31:43.368642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.368512 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"default-dockercfg-pfpf5\"" Nov 01 14:31:43.371308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.371293 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"premium-w0-1f8e2c-kserve-self-signed-certs\"" Nov 01 14:31:43.388699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.388645 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:43.394827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.394800 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82lh6\" (UniqueName: \"kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.394934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.394836 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.394981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.394932 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.394981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.394964 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.395131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.395115 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.395202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.395189 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.495993 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496072 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496100 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82lh6\" (UniqueName: \"kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496118 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496161 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496202 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496602 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496618 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.496802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.496785 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.498400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.498379 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.498678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.498660 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.553315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.553246 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82lh6\" (UniqueName: \"kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6\") pod \"premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.569772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.569743 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:43.579780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.579756 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:43.950852 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:31:43.950820 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cb8da5b_2f76_4b39_b0b2_0dbdf72f83a7.slice/crio-88c9fedaac3ff9922974dbf30bd82a7d2ade4b0cf8ebd4546556c52b1ef4007a WatchSource:0}: Error finding container 88c9fedaac3ff9922974dbf30bd82a7d2ade4b0cf8ebd4546556c52b1ef4007a: Status 404 returned error can't find the container with id 88c9fedaac3ff9922974dbf30bd82a7d2ade4b0cf8ebd4546556c52b1ef4007a Nov 01 14:31:43.953333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:43.953294 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7"] Nov 01 14:31:44.928919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:44.928882 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" event={"ID":"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7","Type":"ContainerStarted","Data":"df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec"} Nov 01 14:31:44.929317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:44.928928 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" event={"ID":"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7","Type":"ContainerStarted","Data":"88c9fedaac3ff9922974dbf30bd82a7d2ade4b0cf8ebd4546556c52b1ef4007a"} Nov 01 14:31:45.044249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:45.044185 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podStartSLOduration=2.044169904 podStartE2EDuration="2.044169904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:31:45.043197136 +0000 UTC m=+844.101348985" watchObservedRunningTime="2026-11-01 14:31:45.044169904 +0000 UTC m=+844.102321754" Nov 01 14:31:45.580200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:45.580166 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:47.185473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.185436 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:47.344567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.344535 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:47.453185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.453106 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c"] Nov 01 14:31:47.456993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.456977 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.509851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.509820 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c"] Nov 01 14:31:47.531435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.531395 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6vx\" (UniqueName: \"kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.531603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.531454 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.531603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.531560 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.556532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.556500 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:31:47.556532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.556526 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:31:47.632641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.632607 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6vx\" (UniqueName: \"kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.632805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.632654 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.632805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.632681 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.633418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.633391 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.635069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.635047 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.689160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.689134 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6vx\" (UniqueName: \"kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx\") pod \"maas-api-revoke-sub-d95ff8b3-dzs6c\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.769934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.769912 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:47.796407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:47.796374 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:48.139814 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:31:48.139785 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ee02a8_2c65_4e05_9cc8_5aa3d56f2b69.slice/crio-6c951e9749fdaa7722983e79c62102bf53076bda7b3588e64086c27ef40eb5d4 WatchSource:0}: Error finding container 6c951e9749fdaa7722983e79c62102bf53076bda7b3588e64086c27ef40eb5d4: Status 404 returned error can't find the container with id 6c951e9749fdaa7722983e79c62102bf53076bda7b3588e64086c27ef40eb5d4 Nov 01 14:31:48.142853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:48.142833 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c"] Nov 01 14:31:48.954505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:48.954474 5473 generic.go:356] "Generic (PLEG): container finished" podID="74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" containerID="4ebcb3023ab5085a871cc38174ebe9f59c2952c4973101b71463635a8e6dfdb7" exitCode=0 Nov 01 14:31:48.954852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:48.954563 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" event={"ID":"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69","Type":"ContainerDied","Data":"4ebcb3023ab5085a871cc38174ebe9f59c2952c4973101b71463635a8e6dfdb7"} Nov 01 14:31:48.954852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:48.954618 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" event={"ID":"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69","Type":"ContainerStarted","Data":"6c951e9749fdaa7722983e79c62102bf53076bda7b3588e64086c27ef40eb5d4"} Nov 01 14:31:50.100205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.100180 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:50.161576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.161548 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx\" (UniqueName: \"kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx\") pod \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " Nov 01 14:31:50.161726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.161703 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca\") pod \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " Nov 01 14:31:50.161828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.161812 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca\") pod \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\" (UID: \"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69\") " Nov 01 14:31:50.162065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.162035 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca" pod "74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" (UID: "74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:31:50.162257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.162207 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:31:50.163733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.163708 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca" pod "74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" (UID: "74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:31:50.163814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.163776 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx" pod "74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" (UID: "74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69"). InnerVolumeSpecName "kube-api-access-sq6vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:31:50.263722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.263698 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:31:50.263870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.263724 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6vx\" (UniqueName: \"kubernetes.io/projected/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69-kube-api-access-sq6vx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:31:50.967706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.967678 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" Nov 01 14:31:50.967875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.967718 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c" event={"ID":"74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69","Type":"ContainerDied","Data":"6c951e9749fdaa7722983e79c62102bf53076bda7b3588e64086c27ef40eb5d4"} Nov 01 14:31:50.967875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:50.967761 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c951e9749fdaa7722983e79c62102bf53076bda7b3588e64086c27ef40eb5d4" Nov 01 14:31:52.257915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:52.257879 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:53.580034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:53.579977 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:53.592501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:53.592469 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:53.997626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:53.997539 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:31:56.602058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:56.602026 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:31:59.202414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:31:59.202378 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:01.005449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:01.005415 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:01.107141 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:32:01.107087 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:32:03.719407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.719375 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg"] Nov 01 14:32:03.719849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.719837 5473 cpu_manager.go:414] "containerMap: removing container" podUID="74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" containerName="revoke-keys" Nov 01 14:32:03.719892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.719850 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" containerName="revoke-keys" Nov 01 14:32:03.719926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.719920 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" containerName="revoke-keys" Nov 01 14:32:03.723088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.723074 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.776663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.776633 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg"] Nov 01 14:32:03.800480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.800453 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.800624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.800524 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.800624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.800600 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.800727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.800634 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.800785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.800764 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.800855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.800843 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72j6h\" (UniqueName: \"kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.820144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.820105 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:03.823904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.823886 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"openshift-service-ca.crt\"" Nov 01 14:32:03.824021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.823887 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"kube-root-ca.crt\"" Nov 01 14:32:03.826389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.826375 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"default-dockercfg-czjqm\"" Nov 01 14:32:03.826446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.826376 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-00a787\"/\"premium-w1-00a787-kserve-self-signed-certs\"" Nov 01 14:32:03.902258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902201 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72j6h\" (UniqueName: \"kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902276 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902324 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902361 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902398 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902463 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902735 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902808 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.902868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.902833 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.904566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.904543 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.904822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.904804 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:03.958352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:03.958330 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72j6h\" (UniqueName: \"kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h\") pod \"premium-w1-00a787-kserve-54c6fc6f86-rvclg\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:04.034835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:04.034806 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:04.410938 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:04.410910 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47537040_036d_45ff_88fa_e059edbaa5aa.slice/crio-ac72d88269dce2d17876636acd2f0a818513b8793a5e59fe1b67d9c84cd39376 WatchSource:0}: Error finding container ac72d88269dce2d17876636acd2f0a818513b8793a5e59fe1b67d9c84cd39376: Status 404 returned error can't find the container with id ac72d88269dce2d17876636acd2f0a818513b8793a5e59fe1b67d9c84cd39376 Nov 01 14:32:04.413509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:04.413487 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg"] Nov 01 14:32:05.023078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.023037 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7"] Nov 01 14:32:05.027222 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.027196 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.063567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.063535 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" event={"ID":"47537040-036d-45ff-88fa-e059edbaa5aa","Type":"ContainerStarted","Data":"9a2b0a70ffa17a078add9ab153c7705186ed51127e3a670961703b2e9f157670"} Nov 01 14:32:05.063567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.063573 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" event={"ID":"47537040-036d-45ff-88fa-e059edbaa5aa","Type":"ContainerStarted","Data":"ac72d88269dce2d17876636acd2f0a818513b8793a5e59fe1b67d9c84cd39376"} Nov 01 14:32:05.081934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.081902 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7"] Nov 01 14:32:05.114458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.114429 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.114619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.114497 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.114619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.114580 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4lv\" (UniqueName: \"kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.125796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.125779 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:32:05.125874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.125824 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:32:05.215757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.215727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.215957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.215804 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.215957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.215887 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4lv\" (UniqueName: \"kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.216644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.216619 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.218602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.218579 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.253561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.253519 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podStartSLOduration=2.253506932 podStartE2EDuration="2.253506932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:05.251675448 +0000 UTC m=+864.309827299" watchObservedRunningTime="2026-11-01 14:32:05.253506932 +0000 UTC m=+864.311658777" Nov 01 14:32:05.272885 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.272859 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4lv\" (UniqueName: \"kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv\") pod \"maas-api-revoke-sub-7db58276-84dl7\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.340931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.340871 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:05.357566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.357539 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:05.715713 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:05.715680 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66666d98_3d16_4dad_8767_38cb9471b07e.slice/crio-582d54114c49350a8690913a16364a36b9c6db82652306d99728f407ae029019 WatchSource:0}: Error finding container 582d54114c49350a8690913a16364a36b9c6db82652306d99728f407ae029019: Status 404 returned error can't find the container with id 582d54114c49350a8690913a16364a36b9c6db82652306d99728f407ae029019 Nov 01 14:32:05.719258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:05.719234 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7"] Nov 01 14:32:06.035877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:06.035853 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:06.071281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:06.071239 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" event={"ID":"66666d98-3d16-4dad-8767-38cb9471b07e","Type":"ContainerStarted","Data":"9fadbe28eb20a0bfc74457cb48bcdc829f39930e26ea529f6ac72a02a4031002"} Nov 01 14:32:06.071411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:06.071290 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" event={"ID":"66666d98-3d16-4dad-8767-38cb9471b07e","Type":"ContainerStarted","Data":"582d54114c49350a8690913a16364a36b9c6db82652306d99728f407ae029019"} Nov 01 14:32:06.205030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:06.204978 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" podStartSLOduration=2.204965055 podStartE2EDuration="2.204965055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:06.203409073 +0000 UTC m=+865.261560953" watchObservedRunningTime="2026-11-01 14:32:06.204965055 +0000 UTC m=+865.263116905" Nov 01 14:32:07.079573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:07.079538 5473 generic.go:356] "Generic (PLEG): container finished" podID="66666d98-3d16-4dad-8767-38cb9471b07e" containerID="9fadbe28eb20a0bfc74457cb48bcdc829f39930e26ea529f6ac72a02a4031002" exitCode=0 Nov 01 14:32:07.081196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:07.081173 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" event={"ID":"66666d98-3d16-4dad-8767-38cb9471b07e","Type":"ContainerDied","Data":"9fadbe28eb20a0bfc74457cb48bcdc829f39930e26ea529f6ac72a02a4031002"} Nov 01 14:32:08.223578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.223557 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:08.349907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.349832 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca\") pod \"66666d98-3d16-4dad-8767-38cb9471b07e\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " Nov 01 14:32:08.350069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.349914 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca\") pod \"66666d98-3d16-4dad-8767-38cb9471b07e\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " Nov 01 14:32:08.350069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.349961 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv\" (UniqueName: \"kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv\") pod \"66666d98-3d16-4dad-8767-38cb9471b07e\" (UID: \"66666d98-3d16-4dad-8767-38cb9471b07e\") " Nov 01 14:32:08.350364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.350337 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca" pod "66666d98-3d16-4dad-8767-38cb9471b07e" (UID: "66666d98-3d16-4dad-8767-38cb9471b07e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:32:08.351977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.351943 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv" pod "66666d98-3d16-4dad-8767-38cb9471b07e" (UID: "66666d98-3d16-4dad-8767-38cb9471b07e"). InnerVolumeSpecName "kube-api-access-wd4lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:08.352110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.352074 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca" pod "66666d98-3d16-4dad-8767-38cb9471b07e" (UID: "66666d98-3d16-4dad-8767-38cb9471b07e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:32:08.451726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.451698 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:08.451726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.451724 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4lv\" (UniqueName: \"kubernetes.io/projected/66666d98-3d16-4dad-8767-38cb9471b07e-kube-api-access-wd4lv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:08.451899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:08.451735 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66666d98-3d16-4dad-8767-38cb9471b07e-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:09.096674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:09.096648 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" Nov 01 14:32:09.096798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:09.096648 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7" event={"ID":"66666d98-3d16-4dad-8767-38cb9471b07e","Type":"ContainerDied","Data":"582d54114c49350a8690913a16364a36b9c6db82652306d99728f407ae029019"} Nov 01 14:32:09.096836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:09.096795 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="582d54114c49350a8690913a16364a36b9c6db82652306d99728f407ae029019" Nov 01 14:32:09.786011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:09.785972 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:14.035137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.035093 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:14.047776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.047753 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:14.137232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.137187 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:32:14.154323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.154295 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b"] Nov 01 14:32:14.154822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.154802 5473 cpu_manager.go:414] "containerMap: removing container" podUID="66666d98-3d16-4dad-8767-38cb9471b07e" containerName="revoke-keys" Nov 01 14:32:14.154822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.154822 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66666d98-3d16-4dad-8767-38cb9471b07e" containerName="revoke-keys" Nov 01 14:32:14.154984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.154957 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="66666d98-3d16-4dad-8767-38cb9471b07e" containerName="revoke-keys" Nov 01 14:32:14.158434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.158414 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.198374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.198349 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:14.210861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.210829 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1f8e2c-openshift-default-dockercfg-cq5wp\"" Nov 01 14:32:14.219090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.219061 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b"] Nov 01 14:32:14.316816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.316736 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.316816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.316792 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.316878 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.316931 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p687\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.317001 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.317059 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.317089 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.317119 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.317176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.317157 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.417878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.417849 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p687\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.417894 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418013 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418053 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418101 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418134 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418264 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418315 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418357 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418570 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418642 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418870 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418901 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.418983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.418942 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.420706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.420678 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.420849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.420833 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.474951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.474925 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.475208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.475189 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p687\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:14.770895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:14.770862 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:15.225504 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:15.225475 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d2aaa9_1674_45ac_8a9d_c02a4649535f.slice/crio-a6d3252eba034a318b224f75c4a76a1f517311e10cbf666f481d2c957d8db5af WatchSource:0}: Error finding container a6d3252eba034a318b224f75c4a76a1f517311e10cbf666f481d2c957d8db5af: Status 404 returned error can't find the container with id a6d3252eba034a318b224f75c4a76a1f517311e10cbf666f481d2c957d8db5af Nov 01 14:32:15.227777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:15.227749 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:32:15.227850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:15.227811 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:32:15.227850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:15.227838 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:32:15.228581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:15.228562 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b"] Nov 01 14:32:16.156045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.155984 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" event={"ID":"d6d2aaa9-1674-45ac-8a9d-c02a4649535f","Type":"ContainerStarted","Data":"b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9"} Nov 01 14:32:16.156276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.156063 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" event={"ID":"d6d2aaa9-1674-45ac-8a9d-c02a4649535f","Type":"ContainerStarted","Data":"a6d3252eba034a318b224f75c4a76a1f517311e10cbf666f481d2c957d8db5af"} Nov 01 14:32:16.292695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.292663 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:16.322016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.321965 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podStartSLOduration=2.321951903 podStartE2EDuration="2.321951903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:16.321727331 +0000 UTC m=+875.379879179" watchObservedRunningTime="2026-11-01 14:32:16.321951903 +0000 UTC m=+875.380103751" Nov 01 14:32:16.771603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.771566 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:16.773102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.773079 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 14:32:16.773188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:16.773141 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 14:32:17.771982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:17.771934 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 14:32:17.772687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:17.772655 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 14:32:18.694626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.694594 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:18.771721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.771686 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 14:32:18.771896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.771746 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 14:32:18.950064 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.949968 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh"] Nov 01 14:32:18.954796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.954774 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972376 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972421 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972534 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972612 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972652 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972729 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972777 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972804 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:18.972857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:18.972835 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlch2\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.007147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.007125 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5b0dcc-openshift-default-dockercfg-4d7r2\"" Nov 01 14:32:19.061839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.061803 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh"] Nov 01 14:32:19.074231 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074191 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074243 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074272 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlch2\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074368 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074434 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074469 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074507 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074533 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074619 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074704 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.074838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.074799 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.075122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.075058 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.075380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.075356 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.075513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.075496 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.077358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.077336 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.077547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.077530 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.151210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.151176 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlch2\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.151541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.151520 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.270033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.270006 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:19.692121 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:19.692092 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce6f30b_c719_4010_a12b_6ef7de59940c.slice/crio-9533e1bd66a7e34b1547726996d080fbe4f6a5af413ce3fd8c94f5f433bd708a WatchSource:0}: Error finding container 9533e1bd66a7e34b1547726996d080fbe4f6a5af413ce3fd8c94f5f433bd708a: Status 404 returned error can't find the container with id 9533e1bd66a7e34b1547726996d080fbe4f6a5af413ce3fd8c94f5f433bd708a Nov 01 14:32:19.694701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.694662 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:32:19.694805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.694747 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:32:19.694805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.694789 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:32:19.695535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.695512 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh"] Nov 01 14:32:19.772063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.772035 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 14:32:19.772192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:19.772081 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 14:32:20.193286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.193252 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" event={"ID":"7ce6f30b-c719-4010-a12b-6ef7de59940c","Type":"ContainerStarted","Data":"80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031"} Nov 01 14:32:20.193673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.193334 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" event={"ID":"7ce6f30b-c719-4010-a12b-6ef7de59940c","Type":"ContainerStarted","Data":"9533e1bd66a7e34b1547726996d080fbe4f6a5af413ce3fd8c94f5f433bd708a"} Nov 01 14:32:20.277336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.277305 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:20.279283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.279021 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 14:32:20.279283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.279070 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 14:32:20.317384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.317339 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podStartSLOduration=2.317326928 podStartE2EDuration="2.317326928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:20.31453931 +0000 UTC m=+879.372691158" watchObservedRunningTime="2026-11-01 14:32:20.317326928 +0000 UTC m=+879.375478777" Nov 01 14:32:20.774675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.774636 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 14:32:20.774963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:20.774933 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 14:32:21.270666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:21.270630 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 14:32:21.271303 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:21.271271 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 14:32:21.771228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:21.771189 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 01 14:32:21.771381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:21.771257 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 01 14:32:22.270511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:22.270479 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 14:32:22.270683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:22.270535 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 14:32:22.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:22.775935 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:22.776329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:22.776309 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:22.776962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:22.776939 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 14:32:23.102422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:23.102336 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:23.271030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:23.270996 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 14:32:23.271457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:23.271049 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 14:32:24.270849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:24.270813 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 14:32:24.271005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:24.270876 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 14:32:25.270987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:25.270952 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 01 14:32:25.271386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:25.271007 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 01 14:32:26.323598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:26.323457 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:26.324118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:26.323866 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:26.324650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:26.324623 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:32:27.506233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.506187 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:27.851028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.850992 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g"] Nov 01 14:32:27.854836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.854806 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:27.906325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.906301 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g"] Nov 01 14:32:27.953650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.953625 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:32:27.953784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.953711 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:32:27.994696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.994665 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:27.994828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.994719 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9v9f\" (UniqueName: \"kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:27.994905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:27.994885 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.095413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.095384 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.095559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.095428 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9v9f\" (UniqueName: \"kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.095559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.095496 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.096049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.096027 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.097712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.097693 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.152278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.152198 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9v9f\" (UniqueName: \"kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f\") pod \"maas-api-revoke-sub-d5a5f8be-56c6g\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.169481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.169458 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:28.533050 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:28.533017 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb0c20b_3ca8_4ce0_9370_65035800d1d2.slice/crio-dd00ea0bee1bd67bd119b68eea4f8ee890283792f8245f8b3a41130622d9974d WatchSource:0}: Error finding container dd00ea0bee1bd67bd119b68eea4f8ee890283792f8245f8b3a41130622d9974d: Status 404 returned error can't find the container with id dd00ea0bee1bd67bd119b68eea4f8ee890283792f8245f8b3a41130622d9974d Nov 01 14:32:28.535849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:28.535824 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g"] Nov 01 14:32:29.278960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:29.278923 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" event={"ID":"ecb0c20b-3ca8-4ce0-9370-65035800d1d2","Type":"ContainerStarted","Data":"da1d1b4209fafd15cb968e13997b985dee4a134d976804a98de23e4144360404"} Nov 01 14:32:29.279154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:29.278974 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" event={"ID":"ecb0c20b-3ca8-4ce0-9370-65035800d1d2","Type":"ContainerStarted","Data":"dd00ea0bee1bd67bd119b68eea4f8ee890283792f8245f8b3a41130622d9974d"} Nov 01 14:32:29.395949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:29.395891 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" podStartSLOduration=2.395873108 podStartE2EDuration="2.395873108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:29.394752702 +0000 UTC m=+888.452904550" watchObservedRunningTime="2026-11-01 14:32:29.395873108 +0000 UTC m=+888.454024958" Nov 01 14:32:30.285979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.285948 5473 generic.go:356] "Generic (PLEG): container finished" podID="ecb0c20b-3ca8-4ce0-9370-65035800d1d2" containerID="da1d1b4209fafd15cb968e13997b985dee4a134d976804a98de23e4144360404" exitCode=0 Nov 01 14:32:30.286373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.286039 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" event={"ID":"ecb0c20b-3ca8-4ce0-9370-65035800d1d2","Type":"ContainerDied","Data":"da1d1b4209fafd15cb968e13997b985dee4a134d976804a98de23e4144360404"} Nov 01 14:32:30.441336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.441304 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn"] Nov 01 14:32:30.445246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.445198 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.503378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.503352 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn"] Nov 01 14:32:30.522507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.522478 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.522651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.522593 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.522651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.522627 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlwz6\" (UniqueName: \"kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.624195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.624123 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.624195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.624172 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.624195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.624190 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlwz6\" (UniqueName: \"kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.624871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.624850 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.626524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.626503 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.680854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.680825 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlwz6\" (UniqueName: \"kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6\") pod \"maas-api-revoke-sub-b1255447-mghzn\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:30.759105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:30.759081 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:31.106601 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:32:31.106570 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:32:31.125834 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:31.125804 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c60fbf3_3f00_46bc_bd7c_d68f3aa36d7f.slice/crio-3d1480b8f072cb4c3ab84a2032a4424091ad6e7a4f1835060f09dc11517c2f78 WatchSource:0}: Error finding container 3d1480b8f072cb4c3ab84a2032a4424091ad6e7a4f1835060f09dc11517c2f78: Status 404 returned error can't find the container with id 3d1480b8f072cb4c3ab84a2032a4424091ad6e7a4f1835060f09dc11517c2f78 Nov 01 14:32:31.129163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.129139 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn"] Nov 01 14:32:31.293836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.293805 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" event={"ID":"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f","Type":"ContainerStarted","Data":"3d1480b8f072cb4c3ab84a2032a4424091ad6e7a4f1835060f09dc11517c2f78"} Nov 01 14:32:31.419715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.419691 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:31.531708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.531676 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f\" (UniqueName: \"kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f\") pod \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " Nov 01 14:32:31.531920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.531763 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca\") pod \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " Nov 01 14:32:31.532029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.531935 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca\") pod \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\" (UID: \"ecb0c20b-3ca8-4ce0-9370-65035800d1d2\") " Nov 01 14:32:31.532716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.532688 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca" pod "ecb0c20b-3ca8-4ce0-9370-65035800d1d2" (UID: "ecb0c20b-3ca8-4ce0-9370-65035800d1d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:32:31.534679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.534652 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f" pod "ecb0c20b-3ca8-4ce0-9370-65035800d1d2" (UID: "ecb0c20b-3ca8-4ce0-9370-65035800d1d2"). InnerVolumeSpecName "kube-api-access-x9v9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:31.536408 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.536381 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca" pod "ecb0c20b-3ca8-4ce0-9370-65035800d1d2" (UID: "ecb0c20b-3ca8-4ce0-9370-65035800d1d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:32:31.633066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.633041 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9v9f\" (UniqueName: \"kubernetes.io/projected/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-kube-api-access-x9v9f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:31.633066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.633068 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:31.633246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.633078 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb0c20b-3ca8-4ce0-9370-65035800d1d2-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:31.899269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:31.899229 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:32.301237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:32.301180 5473 generic.go:356] "Generic (PLEG): container finished" podID="8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" containerID="e56eb7cb78c4ca4ac9f947408511f59e631248108d3db8c830726dc826370b79" exitCode=0 Nov 01 14:32:32.301610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:32.301254 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" event={"ID":"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f","Type":"ContainerDied","Data":"e56eb7cb78c4ca4ac9f947408511f59e631248108d3db8c830726dc826370b79"} Nov 01 14:32:32.302638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:32.302622 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" Nov 01 14:32:32.302736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:32.302622 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g" event={"ID":"ecb0c20b-3ca8-4ce0-9370-65035800d1d2","Type":"ContainerDied","Data":"dd00ea0bee1bd67bd119b68eea4f8ee890283792f8245f8b3a41130622d9974d"} Nov 01 14:32:32.302778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:32.302734 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd00ea0bee1bd67bd119b68eea4f8ee890283792f8245f8b3a41130622d9974d" Nov 01 14:32:33.452528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.452501 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:33.552152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.552115 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca\") pod \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " Nov 01 14:32:33.552337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.552189 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6\" (UniqueName: \"kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6\") pod \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " Nov 01 14:32:33.552337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.552267 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca\") pod \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\" (UID: \"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f\") " Nov 01 14:32:33.552517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.552491 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca" pod "8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" (UID: "8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:32:33.552710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.552695 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:33.554317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.554295 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6" pod "8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" (UID: "8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f"). InnerVolumeSpecName "kube-api-access-rlwz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:33.554386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.554345 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca" pod "8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" (UID: "8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:32:33.653570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.653536 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlwz6\" (UniqueName: \"kubernetes.io/projected/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-kube-api-access-rlwz6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:33.653570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:33.653566 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:34.316212 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:34.316174 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" event={"ID":"8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f","Type":"ContainerDied","Data":"3d1480b8f072cb4c3ab84a2032a4424091ad6e7a4f1835060f09dc11517c2f78"} Nov 01 14:32:34.316381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:34.316226 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn" Nov 01 14:32:34.316381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:34.316249 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d1480b8f072cb4c3ab84a2032a4424091ad6e7a4f1835060f09dc11517c2f78" Nov 01 14:32:36.284899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:36.284871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:40.700696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:40.700659 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:41.193373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:41.193343 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:32:41.195729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:41.195707 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:32:43.345505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.345469 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w"] Nov 01 14:32:43.345955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.345942 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ecb0c20b-3ca8-4ce0-9370-65035800d1d2" containerName="revoke-keys" Nov 01 14:32:43.346001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.345956 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb0c20b-3ca8-4ce0-9370-65035800d1d2" containerName="revoke-keys" Nov 01 14:32:43.346001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.345978 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" containerName="revoke-keys" Nov 01 14:32:43.346001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.345983 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" containerName="revoke-keys" Nov 01 14:32:43.346093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.346069 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" containerName="revoke-keys" Nov 01 14:32:43.346093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.346083 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb0c20b-3ca8-4ce0-9370-65035800d1d2" containerName="revoke-keys" Nov 01 14:32:43.350638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.350621 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.402096 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.402064 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w"] Nov 01 14:32:43.448278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.448251 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:32:43.448278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.448250 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:32:43.456613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.456592 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.456698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.456632 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhhrs\" (UniqueName: \"kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.456771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.456750 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.557877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.557845 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhhrs\" (UniqueName: \"kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.558032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.557925 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.558074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.558037 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.558751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.558733 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.560321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.560305 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.614437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.614383 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhhrs\" (UniqueName: \"kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs\") pod \"maas-api-revoke-sub-d0a8cc0e-4vd2w\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:43.663494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:43.663473 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:44.027697 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:44.027667 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f27e14_07db_4419_8ae0_de0611cfa937.slice/crio-d2294c1a21defc38540aeb338067b86ab3e5b03a2bb82b0e5fadea365d284e45 WatchSource:0}: Error finding container d2294c1a21defc38540aeb338067b86ab3e5b03a2bb82b0e5fadea365d284e45: Status 404 returned error can't find the container with id d2294c1a21defc38540aeb338067b86ab3e5b03a2bb82b0e5fadea365d284e45 Nov 01 14:32:44.030600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:44.030577 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w"] Nov 01 14:32:44.376927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:44.376892 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" event={"ID":"b2f27e14-07db-4419-8ae0-de0611cfa937","Type":"ContainerStarted","Data":"0805cc368435ddd291dd367861172d0a96b5655f21e5813bed451ebdd87cc33d"} Nov 01 14:32:44.377267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:44.376953 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" event={"ID":"b2f27e14-07db-4419-8ae0-de0611cfa937","Type":"ContainerStarted","Data":"d2294c1a21defc38540aeb338067b86ab3e5b03a2bb82b0e5fadea365d284e45"} Nov 01 14:32:44.491887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:44.491843 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" podStartSLOduration=1.491829242 podStartE2EDuration="1.491829242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:44.489119376 +0000 UTC m=+903.547271225" watchObservedRunningTime="2026-11-01 14:32:44.491829242 +0000 UTC m=+903.549981090" Nov 01 14:32:45.096771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:45.096740 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:45.384164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:45.384128 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2f27e14-07db-4419-8ae0-de0611cfa937" containerID="0805cc368435ddd291dd367861172d0a96b5655f21e5813bed451ebdd87cc33d" exitCode=0 Nov 01 14:32:45.384596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:45.384236 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" event={"ID":"b2f27e14-07db-4419-8ae0-de0611cfa937","Type":"ContainerDied","Data":"0805cc368435ddd291dd367861172d0a96b5655f21e5813bed451ebdd87cc33d"} Nov 01 14:32:46.534713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.534688 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:46.587990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.587957 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca\") pod \"b2f27e14-07db-4419-8ae0-de0611cfa937\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " Nov 01 14:32:46.588160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.588083 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca\") pod \"b2f27e14-07db-4419-8ae0-de0611cfa937\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " Nov 01 14:32:46.588160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.588122 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs\" (UniqueName: \"kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs\") pod \"b2f27e14-07db-4419-8ae0-de0611cfa937\" (UID: \"b2f27e14-07db-4419-8ae0-de0611cfa937\") " Nov 01 14:32:46.588419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.588397 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca" pod "b2f27e14-07db-4419-8ae0-de0611cfa937" (UID: "b2f27e14-07db-4419-8ae0-de0611cfa937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:32:46.588520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.588499 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:46.590146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.590121 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca" pod "b2f27e14-07db-4419-8ae0-de0611cfa937" (UID: "b2f27e14-07db-4419-8ae0-de0611cfa937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:32:46.590251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.590178 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs" pod "b2f27e14-07db-4419-8ae0-de0611cfa937" (UID: "b2f27e14-07db-4419-8ae0-de0611cfa937"). InnerVolumeSpecName "kube-api-access-dhhrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:46.690047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.689972 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2f27e14-07db-4419-8ae0-de0611cfa937-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:46.690047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:46.690005 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhhrs\" (UniqueName: \"kubernetes.io/projected/b2f27e14-07db-4419-8ae0-de0611cfa937-kube-api-access-dhhrs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:47.398481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:47.398405 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" Nov 01 14:32:47.398481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:47.398426 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w" event={"ID":"b2f27e14-07db-4419-8ae0-de0611cfa937","Type":"ContainerDied","Data":"d2294c1a21defc38540aeb338067b86ab3e5b03a2bb82b0e5fadea365d284e45"} Nov 01 14:32:47.398481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:47.398473 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2294c1a21defc38540aeb338067b86ab3e5b03a2bb82b0e5fadea365d284e45" Nov 01 14:32:47.505930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:47.505897 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5"] Nov 01 14:32:47.506107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:47.506085 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" containerID="cri-o://ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c" gracePeriod=30 Nov 01 14:32:47.526105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:47.526045 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:49.590489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:49.590432 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:50.266773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.266742 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd"] Nov 01 14:32:50.267250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.267237 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2f27e14-07db-4419-8ae0-de0611cfa937" containerName="revoke-keys" Nov 01 14:32:50.267300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.267251 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f27e14-07db-4419-8ae0-de0611cfa937" containerName="revoke-keys" Nov 01 14:32:50.267335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.267322 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f27e14-07db-4419-8ae0-de0611cfa937" containerName="revoke-keys" Nov 01 14:32:50.270883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.270867 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.322568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.322501 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd"] Nov 01 14:32:50.326751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.326722 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.326871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.326788 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.326945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.326878 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp67t\" (UniqueName: \"kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.369453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.369421 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:32:50.369605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.369589 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:32:50.428305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.428278 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.428420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.428325 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.428420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.428372 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp67t\" (UniqueName: \"kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.429095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.429073 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.430635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.430616 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.485139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.485117 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp67t\" (UniqueName: \"kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t\") pod \"maas-api-revoke-sub-c8e4da22-9j7dd\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.583818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.583737 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:50.948655 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:32:50.948629 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db47471_1e23_498a_b586_d62da98f0679.slice/crio-cf3d288e8ed2ddba08646b2472cf96dc7eae59abb6329d7cbbd2efc69e3e7913 WatchSource:0}: Error finding container cf3d288e8ed2ddba08646b2472cf96dc7eae59abb6329d7cbbd2efc69e3e7913: Status 404 returned error can't find the container with id cf3d288e8ed2ddba08646b2472cf96dc7eae59abb6329d7cbbd2efc69e3e7913 Nov 01 14:32:50.952026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:50.951982 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd"] Nov 01 14:32:51.423846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:51.423812 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" event={"ID":"2db47471-1e23-498a-b586-d62da98f0679","Type":"ContainerStarted","Data":"588f6cd30fdc601b436cb3a1ca7a1f3158bacb33d42bbd65300932db9eb1e5ae"} Nov 01 14:32:51.424032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:51.423877 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" event={"ID":"2db47471-1e23-498a-b586-d62da98f0679","Type":"ContainerStarted","Data":"cf3d288e8ed2ddba08646b2472cf96dc7eae59abb6329d7cbbd2efc69e3e7913"} Nov 01 14:32:51.538903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:51.538849 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" podStartSLOduration=1.5388349529999998 podStartE2EDuration="1.538834953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:32:51.537349576 +0000 UTC m=+910.595501425" watchObservedRunningTime="2026-11-01 14:32:51.538834953 +0000 UTC m=+910.596986804" Nov 01 14:32:52.431666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.431630 5473 generic.go:356] "Generic (PLEG): container finished" podID="2db47471-1e23-498a-b586-d62da98f0679" containerID="588f6cd30fdc601b436cb3a1ca7a1f3158bacb33d42bbd65300932db9eb1e5ae" exitCode=0 Nov 01 14:32:52.432173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.431718 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" event={"ID":"2db47471-1e23-498a-b586-d62da98f0679","Type":"ContainerDied","Data":"588f6cd30fdc601b436cb3a1ca7a1f3158bacb33d42bbd65300932db9eb1e5ae"} Nov 01 14:32:52.761204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.761175 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:32:52.854125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854091 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854147 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854259 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854282 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854303 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854329 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854364 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854403 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854462 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs\") pod \"4de66753-3f67-4f73-b5dd-514858690bc7\" (UID: \"4de66753-3f67-4f73-b5dd-514858690bc7\") " Nov 01 14:32:52.854658 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854396 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:32:52.854863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854804 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:32:52.854863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854438 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:32:52.854863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854596 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:32:52.854863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854838 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:32:52.855155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854967 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.855155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.854986 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4de66753-3f67-4f73-b5dd-514858690bc7-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.855155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.855001 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.855155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.855015 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.855155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.855028 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.856533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.856506 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:32:52.856659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.856623 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:52.856720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.856676 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:32:52.856830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.856813 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg" pod "4de66753-3f67-4f73-b5dd-514858690bc7" (UID: "4de66753-3f67-4f73-b5dd-514858690bc7"). InnerVolumeSpecName "kube-api-access-swjjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:52.956602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.956570 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swjjg\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-kube-api-access-swjjg\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.956602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.956598 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4de66753-3f67-4f73-b5dd-514858690bc7-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.956602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.956609 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4de66753-3f67-4f73-b5dd-514858690bc7-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:52.956602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:52.956617 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4de66753-3f67-4f73-b5dd-514858690bc7-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:53.438926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.438844 5473 generic.go:356] "Generic (PLEG): container finished" podID="4de66753-3f67-4f73-b5dd-514858690bc7" containerID="ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c" exitCode=0 Nov 01 14:32:53.438926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.438897 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" Nov 01 14:32:53.439401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.438926 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" event={"ID":"4de66753-3f67-4f73-b5dd-514858690bc7","Type":"ContainerDied","Data":"ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c"} Nov 01 14:32:53.439401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.438979 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5" event={"ID":"4de66753-3f67-4f73-b5dd-514858690bc7","Type":"ContainerDied","Data":"46834cdb2d5f8c5ff8bcec524e48031cb17da08a44993c244691b8b77ed38c6b"} Nov 01 14:32:53.439401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.439001 5473 scope.go:122] "RemoveContainer" containerID="ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c" Nov 01 14:32:53.449253 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.449209 5473 scope.go:122] "RemoveContainer" containerID="ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c" Nov 01 14:32:53.449499 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:32:53.449483 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c\": container with ID starting with ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c not found: ID does not exist" containerID="ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c" Nov 01 14:32:53.449546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.449507 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c"} err="failed to get container status \"ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c\": rpc error: code = NotFound desc = could not find container \"ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c\": container with ID starting with ab58bc0b85c84123efdb6228b78175b8728c55093af57ab9bb61f28d80d9793c not found: ID does not exist" Nov 01 14:32:53.579839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.579815 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:53.605529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.605499 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5"] Nov 01 14:32:53.655439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.655412 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-nhjm5"] Nov 01 14:32:53.664308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.664287 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca\") pod \"2db47471-1e23-498a-b586-d62da98f0679\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " Nov 01 14:32:53.664399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.664343 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t\" (UniqueName: \"kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t\") pod \"2db47471-1e23-498a-b586-d62da98f0679\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " Nov 01 14:32:53.664461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.664440 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca\") pod \"2db47471-1e23-498a-b586-d62da98f0679\" (UID: \"2db47471-1e23-498a-b586-d62da98f0679\") " Nov 01 14:32:53.664682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.664658 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca" pod "2db47471-1e23-498a-b586-d62da98f0679" (UID: "2db47471-1e23-498a-b586-d62da98f0679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:32:53.666425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.666399 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca" pod "2db47471-1e23-498a-b586-d62da98f0679" (UID: "2db47471-1e23-498a-b586-d62da98f0679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:32:53.666509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.666487 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t" pod "2db47471-1e23-498a-b586-d62da98f0679" (UID: "2db47471-1e23-498a-b586-d62da98f0679"). InnerVolumeSpecName "kube-api-access-bp67t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:32:53.765408 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.765369 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2db47471-1e23-498a-b586-d62da98f0679-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:53.765408 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.765409 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp67t\" (UniqueName: \"kubernetes.io/projected/2db47471-1e23-498a-b586-d62da98f0679-kube-api-access-bp67t\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:53.765596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:53.765419 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2db47471-1e23-498a-b586-d62da98f0679-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:32:54.016117 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:54.015997 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:54.450314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:54.448344 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" event={"ID":"2db47471-1e23-498a-b586-d62da98f0679","Type":"ContainerDied","Data":"cf3d288e8ed2ddba08646b2472cf96dc7eae59abb6329d7cbbd2efc69e3e7913"} Nov 01 14:32:54.450314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:54.448455 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf3d288e8ed2ddba08646b2472cf96dc7eae59abb6329d7cbbd2efc69e3e7913" Nov 01 14:32:54.450314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:54.448546 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd" Nov 01 14:32:55.081904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:55.081868 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" path="/var/lib/kubelet/pods/4de66753-3f67-4f73-b5dd-514858690bc7/volumes" Nov 01 14:32:55.988273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:55.988161 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:56.157499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:56.157452 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:58.499794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:58.499763 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:32:59.897664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.897633 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl"] Nov 01 14:32:59.898299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.898280 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2db47471-1e23-498a-b586-d62da98f0679" containerName="revoke-keys" Nov 01 14:32:59.898388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.898301 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2db47471-1e23-498a-b586-d62da98f0679" containerName="revoke-keys" Nov 01 14:32:59.898388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.898351 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" Nov 01 14:32:59.898388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.898359 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" Nov 01 14:32:59.898554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.898487 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de66753-3f67-4f73-b5dd-514858690bc7" containerName="istio-proxy" Nov 01 14:32:59.898554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.898504 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2db47471-1e23-498a-b586-d62da98f0679" containerName="revoke-keys" Nov 01 14:32:59.901773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.901754 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:32:59.954130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:32:59.954105 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-007270-openshift-default-dockercfg-78vkt\"" Nov 01 14:33:00.006985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.006957 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl"] Nov 01 14:33:00.035305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035279 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035334 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035458 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtl5\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035523 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035547 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035573 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035650 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035721 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.035778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.035766 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136348 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136406 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136453 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136589 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136694 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136654 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136735 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtl5\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136789 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136794 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136827 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136870 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.136971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.136946 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.137117 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.137094 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.137657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.137629 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.140251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.137932 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.146755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.146732 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.148050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.147978 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.193803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.193766 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.193943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.193818 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtl5\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.214790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.214766 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:00.627540 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:33:00.627501 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbbd1b0_ba54_4cb3_ac66_58dbee7bf082.slice/crio-4e44bab43dcbb59b7e726a3f51a4b666bd58495657c22c00a6323feb92348317 WatchSource:0}: Error finding container 4e44bab43dcbb59b7e726a3f51a4b666bd58495657c22c00a6323feb92348317: Status 404 returned error can't find the container with id 4e44bab43dcbb59b7e726a3f51a4b666bd58495657c22c00a6323feb92348317 Nov 01 14:33:00.629817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.629776 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:00.629908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.629860 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:00.629908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.629902 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:00.631124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.631103 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl"] Nov 01 14:33:00.912592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.912508 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs"] Nov 01 14:33:00.916442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.916423 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:00.968667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.968628 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs"] Nov 01 14:33:00.968937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.968912 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:33:00.969006 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:00.968966 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:33:01.049047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.049014 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7lzp\" (UniqueName: \"kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.049258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.049075 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.049258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.049232 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.110999 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:33:01.110968 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:33:01.151138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.151092 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.151590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.151568 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7lzp\" (UniqueName: \"kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.152281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.151862 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.152281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.151870 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.154564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.154536 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.208749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.208680 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7lzp\" (UniqueName: \"kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp\") pod \"maas-api-revoke-sub-684ba7ed-zcfhs\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.247697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.247401 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:01.493543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.493458 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" event={"ID":"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082","Type":"ContainerStarted","Data":"56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0"} Nov 01 14:33:01.493543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.493517 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" event={"ID":"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082","Type":"ContainerStarted","Data":"4e44bab43dcbb59b7e726a3f51a4b666bd58495657c22c00a6323feb92348317"} Nov 01 14:33:01.613193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.613144 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podStartSLOduration=2.6131297350000002 podStartE2EDuration="2.613129735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:01.612514405 +0000 UTC m=+920.670666292" watchObservedRunningTime="2026-11-01 14:33:01.613129735 +0000 UTC m=+920.671281585" Nov 01 14:33:01.616141 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:33:01.616114 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafbc8cc_a30d_485a_a77c_b2697c816620.slice/crio-aaec9051605904f43480fbd6541656bcbcad1c34c0ee2730d91dcfe44ab0a77e WatchSource:0}: Error finding container aaec9051605904f43480fbd6541656bcbcad1c34c0ee2730d91dcfe44ab0a77e: Status 404 returned error can't find the container with id aaec9051605904f43480fbd6541656bcbcad1c34c0ee2730d91dcfe44ab0a77e Nov 01 14:33:01.665678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:01.665648 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs"] Nov 01 14:33:02.215308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.215206 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:02.216649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.216623 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 14:33:02.216787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.216675 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 14:33:02.502081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.501992 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" event={"ID":"cafbc8cc-a30d-485a-a77c-b2697c816620","Type":"ContainerStarted","Data":"7ad42e9f7ee82c73b5e2c961ecea43f92cf84dab2181cdd0486dd1c70ec8be07"} Nov 01 14:33:02.502081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.502049 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" event={"ID":"cafbc8cc-a30d-485a-a77c-b2697c816620","Type":"ContainerStarted","Data":"aaec9051605904f43480fbd6541656bcbcad1c34c0ee2730d91dcfe44ab0a77e"} Nov 01 14:33:02.614921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.614869 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" podStartSLOduration=2.614857189 podStartE2EDuration="2.614857189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:02.614470742 +0000 UTC m=+921.672622589" watchObservedRunningTime="2026-11-01 14:33:02.614857189 +0000 UTC m=+921.673009038" Nov 01 14:33:02.950259 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:02.949039 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:03.215360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:03.215278 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 14:33:03.215360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:03.215330 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 14:33:03.510581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:03.510498 5473 generic.go:356] "Generic (PLEG): container finished" podID="cafbc8cc-a30d-485a-a77c-b2697c816620" containerID="7ad42e9f7ee82c73b5e2c961ecea43f92cf84dab2181cdd0486dd1c70ec8be07" exitCode=0 Nov 01 14:33:03.510750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:03.510583 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" event={"ID":"cafbc8cc-a30d-485a-a77c-b2697c816620","Type":"ContainerDied","Data":"7ad42e9f7ee82c73b5e2c961ecea43f92cf84dab2181cdd0486dd1c70ec8be07"} Nov 01 14:33:04.216059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.216025 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 14:33:04.216464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.216073 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 14:33:04.687950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.687927 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:04.793874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.793794 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca\") pod \"cafbc8cc-a30d-485a-a77c-b2697c816620\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " Nov 01 14:33:04.794011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.793970 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp\" (UniqueName: \"kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp\") pod \"cafbc8cc-a30d-485a-a77c-b2697c816620\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " Nov 01 14:33:04.794061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.794041 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca\") pod \"cafbc8cc-a30d-485a-a77c-b2697c816620\" (UID: \"cafbc8cc-a30d-485a-a77c-b2697c816620\") " Nov 01 14:33:04.794188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.794169 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca" pod "cafbc8cc-a30d-485a-a77c-b2697c816620" (UID: "cafbc8cc-a30d-485a-a77c-b2697c816620"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:33:04.794418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.794393 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:33:04.796036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.795987 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca" pod "cafbc8cc-a30d-485a-a77c-b2697c816620" (UID: "cafbc8cc-a30d-485a-a77c-b2697c816620"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:33:04.796117 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.796054 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp" pod "cafbc8cc-a30d-485a-a77c-b2697c816620" (UID: "cafbc8cc-a30d-485a-a77c-b2697c816620"). InnerVolumeSpecName "kube-api-access-t7lzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:33:04.895738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.895710 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7lzp\" (UniqueName: \"kubernetes.io/projected/cafbc8cc-a30d-485a-a77c-b2697c816620-kube-api-access-t7lzp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:33:04.895738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:04.895737 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cafbc8cc-a30d-485a-a77c-b2697c816620-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:33:05.216169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:05.216134 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 14:33:05.216580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:05.216187 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 14:33:05.542475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:05.542439 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" event={"ID":"cafbc8cc-a30d-485a-a77c-b2697c816620","Type":"ContainerDied","Data":"aaec9051605904f43480fbd6541656bcbcad1c34c0ee2730d91dcfe44ab0a77e"} Nov 01 14:33:05.542736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:05.542720 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaec9051605904f43480fbd6541656bcbcad1c34c0ee2730d91dcfe44ab0a77e" Nov 01 14:33:05.542889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:05.542877 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs" Nov 01 14:33:06.215431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:06.215399 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Nov 01 14:33:06.215578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:06.215447 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Nov 01 14:33:07.219281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:07.219253 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:07.219716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:07.219697 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:07.220448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:07.220425 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:33:07.407052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:07.407022 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:09.292145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:09.292112 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:09.457196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:09.457165 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:09.613441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:09.613371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:11.807021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:11.806989 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:16.201790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:16.201757 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:18.699653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.699617 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:18.799896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.799863 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn"] Nov 01 14:33:18.800419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.800404 5473 cpu_manager.go:414] "containerMap: removing container" podUID="cafbc8cc-a30d-485a-a77c-b2697c816620" containerName="revoke-keys" Nov 01 14:33:18.800472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.800420 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cafbc8cc-a30d-485a-a77c-b2697c816620" containerName="revoke-keys" Nov 01 14:33:18.800508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.800494 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cafbc8cc-a30d-485a-a77c-b2697c816620" containerName="revoke-keys" Nov 01 14:33:18.803847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.803832 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:18.859394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.859360 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn"] Nov 01 14:33:18.872094 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.872061 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:18.904409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.904382 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:33:18.904549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.904475 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:33:18.960250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.960150 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:18.960250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.960196 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfgv\" (UniqueName: \"kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:18.960487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:18.960369 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.061697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.061661 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.061896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.061727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfgv\" (UniqueName: \"kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.061896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.061870 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.062856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.062830 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.066586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.066563 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.119269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.119205 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfgv\" (UniqueName: \"kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv\") pod \"maas-api-revoke-sub-ee0ae16a-2r4bn\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.417041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.417009 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:19.784534 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:33:19.784505 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111eb51e_10b3_4384_aa76_b5ae0e322842.slice/crio-2147f4ec53d54a0eeb5a94b5099aed432363cd6bc813b7d3db07707d206f824c WatchSource:0}: Error finding container 2147f4ec53d54a0eeb5a94b5099aed432363cd6bc813b7d3db07707d206f824c: Status 404 returned error can't find the container with id 2147f4ec53d54a0eeb5a94b5099aed432363cd6bc813b7d3db07707d206f824c Nov 01 14:33:19.787182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.787154 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn"] Nov 01 14:33:19.917794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:19.917757 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:20.643973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:20.643941 5473 generic.go:356] "Generic (PLEG): container finished" podID="111eb51e-10b3-4384-aa76-b5ae0e322842" containerID="787bf1b26e416f21a4916be6b446bd2539c73bdacf913a24cab8ca5aea71aa3c" exitCode=0 Nov 01 14:33:20.644126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:20.644013 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" event={"ID":"111eb51e-10b3-4384-aa76-b5ae0e322842","Type":"ContainerDied","Data":"787bf1b26e416f21a4916be6b446bd2539c73bdacf913a24cab8ca5aea71aa3c"} Nov 01 14:33:20.644126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:20.644078 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" event={"ID":"111eb51e-10b3-4384-aa76-b5ae0e322842","Type":"ContainerStarted","Data":"2147f4ec53d54a0eeb5a94b5099aed432363cd6bc813b7d3db07707d206f824c"} Nov 01 14:33:20.906336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:20.906258 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:21.787413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.787391 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:21.789482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.789461 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv\" (UniqueName: \"kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv\") pod \"111eb51e-10b3-4384-aa76-b5ae0e322842\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " Nov 01 14:33:21.789601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.789506 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca\") pod \"111eb51e-10b3-4384-aa76-b5ae0e322842\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " Nov 01 14:33:21.789601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.789564 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca\") pod \"111eb51e-10b3-4384-aa76-b5ae0e322842\" (UID: \"111eb51e-10b3-4384-aa76-b5ae0e322842\") " Nov 01 14:33:21.789948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.789927 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca" pod "111eb51e-10b3-4384-aa76-b5ae0e322842" (UID: "111eb51e-10b3-4384-aa76-b5ae0e322842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:33:21.791456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.791435 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca" pod "111eb51e-10b3-4384-aa76-b5ae0e322842" (UID: "111eb51e-10b3-4384-aa76-b5ae0e322842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:33:21.791569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.791549 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv" pod "111eb51e-10b3-4384-aa76-b5ae0e322842" (UID: "111eb51e-10b3-4384-aa76-b5ae0e322842"). InnerVolumeSpecName "kube-api-access-pjfgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:33:21.890556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.890522 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfgv\" (UniqueName: \"kubernetes.io/projected/111eb51e-10b3-4384-aa76-b5ae0e322842-kube-api-access-pjfgv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:33:21.890556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.890559 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:33:21.890762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:21.890575 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/111eb51e-10b3-4384-aa76-b5ae0e322842-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:33:22.657847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:22.657812 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" event={"ID":"111eb51e-10b3-4384-aa76-b5ae0e322842","Type":"ContainerDied","Data":"2147f4ec53d54a0eeb5a94b5099aed432363cd6bc813b7d3db07707d206f824c"} Nov 01 14:33:22.658280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:22.657864 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2147f4ec53d54a0eeb5a94b5099aed432363cd6bc813b7d3db07707d206f824c" Nov 01 14:33:22.658280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:22.657832 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn" Nov 01 14:33:25.297672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:25.297633 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:26.449784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:26.449751 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:29.734416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:29.734350 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:31.106586 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:33:31.106549 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:33:34.147911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:34.147880 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:38.636532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:38.636438 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:43.148406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:43.148366 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:47.472289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.472190 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:47.677622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.677567 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:47.855397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.855363 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:47.991238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.987681 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6"] Nov 01 14:33:47.991238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.988253 5473 cpu_manager.go:414] "containerMap: removing container" podUID="111eb51e-10b3-4384-aa76-b5ae0e322842" containerName="revoke-keys" Nov 01 14:33:47.991238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.988267 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111eb51e-10b3-4384-aa76-b5ae0e322842" containerName="revoke-keys" Nov 01 14:33:47.991238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.988416 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="111eb51e-10b3-4384-aa76-b5ae0e322842" containerName="revoke-keys" Nov 01 14:33:47.993478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:47.993458 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.039828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.039792 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn"] Nov 01 14:33:48.044719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.044694 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.116830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.116750 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn"] Nov 01 14:33:48.165171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165178 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165198 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.165414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165299 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.165414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165332 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr6g6\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.165414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165349 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165367 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.165584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165562 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165599 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165686 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165734 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165778 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.165824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165799 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.165857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165834 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.165857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.165851 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.166128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.166113 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2w2w\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.166192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.166181 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.166253 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.166225 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.188594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.188563 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6"] Nov 01 14:33:48.266808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266776 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2w2w\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.266808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266819 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266855 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266889 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266922 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266955 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.266985 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267012 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr6g6\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267034 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267058 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267105 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267129 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267185 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267262 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267297 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267325 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267357 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267365 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267362 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267434 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.267523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267529 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.268095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267590 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.268095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267712 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.268095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267722 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.268095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.267742 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.268095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.268014 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.268325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.268289 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.268325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.268312 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.269473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.269446 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.270073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.270057 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.270387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.270366 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.270566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.270551 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.323798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.323765 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.323924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.323859 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.324228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.324196 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr6g6\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-clqh6\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.324272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.324195 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2w2w\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dtfjn\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.358675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.358639 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:48.607563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.607533 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:48.773080 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:33:48.773044 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e6f6f9_8ec9_49c1_9b52_b3fa88a6ebec.slice/crio-6d67ab9ed1733a4edd93fb2083a405f9832e0f87631ff5d0d64fb72f32dc2b81 WatchSource:0}: Error finding container 6d67ab9ed1733a4edd93fb2083a405f9832e0f87631ff5d0d64fb72f32dc2b81: Status 404 returned error can't find the container with id 6d67ab9ed1733a4edd93fb2083a405f9832e0f87631ff5d0d64fb72f32dc2b81 Nov 01 14:33:48.779452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.779418 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn"] Nov 01 14:33:48.779858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.779811 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:48.779976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.779948 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:48.780062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.780045 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:48.833916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.833890 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" event={"ID":"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec","Type":"ContainerStarted","Data":"6d67ab9ed1733a4edd93fb2083a405f9832e0f87631ff5d0d64fb72f32dc2b81"} Nov 01 14:33:48.980560 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:33:48.980524 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7504900e_d79c_40bf_adfc_fabccbdaf908.slice/crio-a183a8e4ecdef528820368b28757cc480bd806ae67972687825e887eddd9d37d WatchSource:0}: Error finding container a183a8e4ecdef528820368b28757cc480bd806ae67972687825e887eddd9d37d: Status 404 returned error can't find the container with id a183a8e4ecdef528820368b28757cc480bd806ae67972687825e887eddd9d37d Nov 01 14:33:48.982192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.982168 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6"] Nov 01 14:33:48.982328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.982298 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:48.982387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.982361 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:48.982443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:48.982390 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:33:49.879236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:49.879187 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" event={"ID":"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec","Type":"ContainerStarted","Data":"5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0"} Nov 01 14:33:49.882993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:49.882961 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" event={"ID":"7504900e-d79c-40bf-adfc-fabccbdaf908","Type":"ContainerStarted","Data":"d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578"} Nov 01 14:33:49.883172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:49.883012 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" event={"ID":"7504900e-d79c-40bf-adfc-fabccbdaf908","Type":"ContainerStarted","Data":"a183a8e4ecdef528820368b28757cc480bd806ae67972687825e887eddd9d37d"} Nov 01 14:33:50.192637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.192540 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podStartSLOduration=3.192518107 podStartE2EDuration="3.192518107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:50.191745966 +0000 UTC m=+969.249897817" watchObservedRunningTime="2026-11-01 14:33:50.192518107 +0000 UTC m=+969.250669957" Nov 01 14:33:50.192770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.192662 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podStartSLOduration=3.19265828 podStartE2EDuration="3.19265828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:33:50.074058124 +0000 UTC m=+969.132209976" watchObservedRunningTime="2026-11-01 14:33:50.19265828 +0000 UTC m=+969.250810131" Nov 01 14:33:50.361332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.361300 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:50.361742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.361715 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:50.361962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.361937 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:50.608422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.608386 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:50.610191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.610162 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:50.610315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:50.610209 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:51.361629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:51.361591 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:51.362677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:51.362645 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:51.608708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:51.608675 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:51.608894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:51.608727 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:52.278570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:52.278534 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:52.368238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:52.366593 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:52.368238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:52.366657 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:52.608671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:52.608590 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:52.608671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:52.608646 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:53.369239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:53.368609 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:53.369239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:53.368675 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:53.608532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:53.608497 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:53.608710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:53.608553 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:54.362656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:54.362616 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:54.362965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:54.362932 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:54.607878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:54.607846 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:54.608403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:54.607903 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:55.360074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:55.360030 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:55.360286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:55.360100 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:55.608660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:55.608620 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:55.609066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:55.608688 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:56.359998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:56.359945 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dtfjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Nov 01 14:33:56.360184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:56.360023 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Nov 01 14:33:56.608322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:56.608282 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 01 14:33:56.608531 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:56.608348 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 01 14:33:56.968659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:56.968339 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:33:57.364007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.363972 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:57.364421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.364399 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:57.365196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.365174 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:33:57.613233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.613189 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:57.613498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.613468 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:57.614250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.614178 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:33:57.667737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:33:57.667685 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:01.107331 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:34:01.107298 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:34:01.335285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:01.335245 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:05.764765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:05.764729 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:10.225614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:10.225575 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:13.459599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:13.459557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:14.693925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:14.693892 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:19.295554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:19.295322 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:19.913834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:19.913796 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x"] Nov 01 14:34:19.918565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:19.918534 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:19.964918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:19.964877 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x"] Nov 01 14:34:20.109053 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.108995 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.109606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.109577 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.109858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.109840 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.110001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.109984 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.110163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.110139 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgw5l\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.110378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.110347 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.110468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.110396 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.110532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.110474 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.110532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.110517 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.211932 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.211995 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212049 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212065 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212104 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgw5l\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212147 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212161 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212249 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212283 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.212653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.212427 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.213280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.213251 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.213403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.213296 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.213403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.213353 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.213498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.213477 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.215096 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.215070 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.215229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.215191 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.281881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.281847 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.282113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.281941 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgw5l\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:20.565499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:20.565445 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:21.096963 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:34:21.096936 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37576851_40f6_42ab_bc50_be00a24bc5e1.slice/crio-d34d4ea6a172eecc151e206fa78ad864c40f00ad1549f58496cef9becb1ecfa4 WatchSource:0}: Error finding container d34d4ea6a172eecc151e206fa78ad864c40f00ad1549f58496cef9becb1ecfa4: Status 404 returned error can't find the container with id d34d4ea6a172eecc151e206fa78ad864c40f00ad1549f58496cef9becb1ecfa4 Nov 01 14:34:21.098563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:21.098534 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x"] Nov 01 14:34:21.099193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:21.099164 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:34:21.099324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:21.099250 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:34:21.099324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:21.099282 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:34:21.120921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:21.120892 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" event={"ID":"37576851-40f6-42ab-bc50-be00a24bc5e1","Type":"ContainerStarted","Data":"d34d4ea6a172eecc151e206fa78ad864c40f00ad1549f58496cef9becb1ecfa4"} Nov 01 14:34:22.154906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:22.154866 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" event={"ID":"37576851-40f6-42ab-bc50-be00a24bc5e1","Type":"ContainerStarted","Data":"d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b"} Nov 01 14:34:22.324154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:22.324102 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podStartSLOduration=3.324088848 podStartE2EDuration="3.324088848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:22.321969726 +0000 UTC m=+1001.380121576" watchObservedRunningTime="2026-11-01 14:34:22.324088848 +0000 UTC m=+1001.382240697" Nov 01 14:34:22.566596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:22.566561 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:22.568072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:22.568046 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 14:34:22.568190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:22.568097 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 14:34:23.566799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:23.566769 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 14:34:23.567176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:23.566821 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 14:34:24.212576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:24.212544 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:24.566309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:24.566274 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 14:34:24.566491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:24.566340 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 14:34:25.566779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:25.566741 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 14:34:25.567298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:25.566793 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 14:34:26.567141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:26.567100 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 14:34:26.567585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:26.567162 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 14:34:27.567175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:27.567133 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 01 14:34:27.567681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:27.567193 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 01 14:34:28.571069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:28.571040 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:28.571525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:28.571503 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:28.572248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:28.572230 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:34:28.852891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:28.852755 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:31.108612 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:34:31.108564 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:34:33.197618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:33.197585 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:35.784928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.784894 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd"] Nov 01 14:34:35.788985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.788969 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.842677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.842646 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd"] Nov 01 14:34:35.887991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.887952 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"e2e-conflict-b-a12d4d90-kserve-self-signed-certs\"" Nov 01 14:34:35.890832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.890807 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.890955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.890896 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhcc9\" (UniqueName: \"kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.890955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.890952 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.891082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.891061 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.891160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.891143 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.891276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.891261 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.948432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.948397 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:35.992501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992468 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.992682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992516 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhcc9\" (UniqueName: \"kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.992682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992563 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.992682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992609 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.992682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992662 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.992909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992742 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.992909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992885 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.993021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.992988 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.993072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.993045 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.994957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.994932 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:35.995203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:35.995186 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:36.049548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:36.049488 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhcc9\" (UniqueName: \"kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9\") pod \"e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:36.101403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:36.101372 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:36.485977 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:34:36.485940 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9f30019_6fdd_4d97_9271_58fa88ff39c7.slice/crio-8d3ce25ecf3e37270e71662d74d8586a95d40ee61345bb1f62f124f8918ec9a4 WatchSource:0}: Error finding container 8d3ce25ecf3e37270e71662d74d8586a95d40ee61345bb1f62f124f8918ec9a4: Status 404 returned error can't find the container with id 8d3ce25ecf3e37270e71662d74d8586a95d40ee61345bb1f62f124f8918ec9a4 Nov 01 14:34:36.490195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:36.490175 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd"] Nov 01 14:34:37.256145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:37.256110 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" event={"ID":"a9f30019-6fdd-4d97-9271-58fa88ff39c7","Type":"ContainerStarted","Data":"fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5"} Nov 01 14:34:37.256145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:37.256152 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" event={"ID":"a9f30019-6fdd-4d97-9271-58fa88ff39c7","Type":"ContainerStarted","Data":"8d3ce25ecf3e37270e71662d74d8586a95d40ee61345bb1f62f124f8918ec9a4"} Nov 01 14:34:37.371311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:37.371183 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" podStartSLOduration=2.3711647940000002 podStartE2EDuration="2.371164794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:34:37.370175553 +0000 UTC m=+1016.428327403" watchObservedRunningTime="2026-11-01 14:34:37.371164794 +0000 UTC m=+1016.429316644" Nov 01 14:34:37.640202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:37.640164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:38.101652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:38.101622 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:42.016650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:42.016610 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:46.102341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:46.102304 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:46.115725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:46.115699 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:46.327431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:46.327400 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:34:46.520935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:46.520910 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:48.115020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:48.114976 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:48.278998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:48.278967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:50.988533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:50.988494 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:55.398366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:55.398332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:34:59.795011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:34:59.794968 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:01.107121 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:35:01.107083 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:35:03.142803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:03.142767 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:03.336924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:03.336896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:03.496044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:03.495966 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:04.394612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:04.394579 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:07.276126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.276092 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc"] Nov 01 14:35:07.280993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.280963 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.338767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.338737 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.338942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.338773 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smbz\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.338942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.338800 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.339068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.338980 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.385161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385119 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 14:35:07.385161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385148 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee13fb54a62a\"" Nov 01 14:35:07.385161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385122 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-04c6b8e2a5aa\"" Nov 01 14:35:07.385434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385132 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-091b76ab0055\"" Nov 01 14:35:07.385434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385134 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vn9m6\"" Nov 01 14:35:07.385434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385122 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 14:35:07.385434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385168 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 14:35:07.385434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.385211 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 14:35:07.388720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.388699 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc"] Nov 01 14:35:07.439926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.439898 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.439926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.439927 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smbz\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.440102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.439961 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.440102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.440026 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.440744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.440720 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.440859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.440800 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.442465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.442446 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.496724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.496697 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smbz\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz\") pod \"payload-processing-external-model-6b765ff78c-ln8xc\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.600017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.599950 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:07.965069 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:35:07.965035 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ec8770_31d5_467e_95fb_2b7c950174d8.slice/crio-61774350ee85e6f64e4675eae8c31a313f6ef26683e015ad6cb7f1db5abbbfaf WatchSource:0}: Error finding container 61774350ee85e6f64e4675eae8c31a313f6ef26683e015ad6cb7f1db5abbbfaf: Status 404 returned error can't find the container with id 61774350ee85e6f64e4675eae8c31a313f6ef26683e015ad6cb7f1db5abbbfaf Nov 01 14:35:07.967993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:07.967968 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc"] Nov 01 14:35:08.475426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:08.475396 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" event={"ID":"d1ec8770-31d5-467e-95fb-2b7c950174d8","Type":"ContainerStarted","Data":"ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc"} Nov 01 14:35:08.475812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:08.475450 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:08.475812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:08.475469 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" event={"ID":"d1ec8770-31d5-467e-95fb-2b7c950174d8","Type":"ContainerStarted","Data":"61774350ee85e6f64e4675eae8c31a313f6ef26683e015ad6cb7f1db5abbbfaf"} Nov 01 14:35:08.591786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:08.591734 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" podStartSLOduration=1.5917205719999998 podStartE2EDuration="1.591720572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:08.588819586 +0000 UTC m=+1047.646971436" watchObservedRunningTime="2026-11-01 14:35:08.591720572 +0000 UTC m=+1047.649872422" Nov 01 14:35:08.775585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:08.775550 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:09.852655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:09.852619 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:13.443241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:13.440325 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:18.037381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:18.037346 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:18.855775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:18.855745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:19.484721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:19.484689 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:35:22.397520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:22.397484 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:26.551590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:26.551556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:26.767339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:26.767308 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:31.215859 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:35:31.215819 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:35:32.778147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:32.778094 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:37.309919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:37.309887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:41.315931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:41.315902 5473 scope.go:122] "RemoveContainer" containerID="808c030a2ca3a4a61162129cae649ac038de1c0e9b2bed4788c970c418421460" Nov 01 14:35:41.325719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:41.325696 5473 scope.go:122] "RemoveContainer" containerID="2d2d0fca32d2e3125fbe803ae9f943388c27d6eceabab56674e4396ebd4568af" Nov 01 14:35:41.334960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:41.334936 5473 scope.go:122] "RemoveContainer" containerID="403ba6c790003740e8e932eadd2c0e0662147b691592e513a6a25e50103a182b" Nov 01 14:35:41.343291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:41.343269 5473 scope.go:122] "RemoveContainer" containerID="24df77910c2fc699a61daa711d542d87c94d2e074dbc073739fec33e1b07900f" Nov 01 14:35:41.351578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:41.351562 5473 scope.go:122] "RemoveContainer" containerID="20cb51a07698200f011601e1544adcc7ddf5f485e66188b5b648c3ae743e0312" Nov 01 14:35:41.687484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:41.687392 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:46.143371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:46.143332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:46.358697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:46.358661 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:48.436897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:48.436858 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:50.488878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:50.488795 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:54.900888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:54.900852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:56.654892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.654857 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z"] Nov 01 14:35:56.658963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.658946 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.755887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.755854 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z"] Nov 01 14:35:56.770432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770400 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770437 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770482 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770566 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770608 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770633 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gw7s\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770654 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770795 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.770977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.770821 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872088 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872127 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872169 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872195 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872240 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gw7s\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872450 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872540 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872574 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872693 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872712 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.872928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.872910 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.873014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.873000 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.873332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.873312 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.874673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.874644 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.875507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.875485 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.929311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.929240 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.929466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.929377 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gw7s\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:56.972494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:56.972462 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:57.392067 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:35:57.392039 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d7eb3e_40d3_403a_b8dd_9c75c167848f.slice/crio-1466ee603dfe2b7fc1c42294caeaed02575ea4f840fc95c02f180f367c6ae56c WatchSource:0}: Error finding container 1466ee603dfe2b7fc1c42294caeaed02575ea4f840fc95c02f180f367c6ae56c: Status 404 returned error can't find the container with id 1466ee603dfe2b7fc1c42294caeaed02575ea4f840fc95c02f180f367c6ae56c Nov 01 14:35:57.394506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.394473 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:35:57.394610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.394539 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:35:57.394610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.394567 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:35:57.394933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.394911 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z"] Nov 01 14:35:57.888320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.888280 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" event={"ID":"20d7eb3e-40d3-403a-b8dd-9c75c167848f","Type":"ContainerStarted","Data":"a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f"} Nov 01 14:35:57.888750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.888351 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" event={"ID":"20d7eb3e-40d3-403a-b8dd-9c75c167848f","Type":"ContainerStarted","Data":"1466ee603dfe2b7fc1c42294caeaed02575ea4f840fc95c02f180f367c6ae56c"} Nov 01 14:35:57.973471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:57.973444 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:35:58.008056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:58.008008 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podStartSLOduration=2.007994626 podStartE2EDuration="2.007994626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:35:58.005210341 +0000 UTC m=+1097.063362191" watchObservedRunningTime="2026-11-01 14:35:58.007994626 +0000 UTC m=+1097.066146486" Nov 01 14:35:58.974320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:58.974283 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 14:35:58.974718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:58.974344 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 14:35:59.630663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:59.630626 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:35:59.973143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:59.973061 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 14:35:59.973143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:35:59.973109 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 14:36:00.973551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:00.973511 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 14:36:00.973946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:00.973587 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 14:36:01.107036 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:36:01.107003 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:36:01.972802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:01.972764 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 14:36:01.972974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:01.972818 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 14:36:02.973489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:02.973402 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 14:36:02.973489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:02.973462 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 14:36:03.972736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:03.972703 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 01 14:36:03.972916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:03.972762 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 01 14:36:04.128774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:04.128740 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:04.977049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:04.977013 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:36:04.977454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:04.977433 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:36:04.978252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:04.978211 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:36:08.610944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:08.610904 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:09.308001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:09.307970 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd"] Nov 01 14:36:09.339872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:09.339841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:13.034070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.034031 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7"] Nov 01 14:36:13.048993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.048956 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.072913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.072881 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:13.073366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.073345 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.073464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.073452 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtmrf\" (UniqueName: \"kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.073532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.073514 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.090750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.090715 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7"] Nov 01 14:36:13.133456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.133428 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:36:13.133621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.133545 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:36:13.174307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.174279 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtmrf\" (UniqueName: \"kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.174485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.174318 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.174485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.174459 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.175105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.175086 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.176904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.176879 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.231962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.231919 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtmrf\" (UniqueName: \"kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf\") pod \"maas-api-revoke-sub-aeb09241-9mdx7\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.364766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.364694 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:13.734656 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:13.734623 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af05ded_ba6c_425b_aa00_6328321ef265.slice/crio-3ff157a5a9a95327ba8444f3b52de84ae461659fe1ca9f6e3ebd116f7b28f7ff WatchSource:0}: Error finding container 3ff157a5a9a95327ba8444f3b52de84ae461659fe1ca9f6e3ebd116f7b28f7ff: Status 404 returned error can't find the container with id 3ff157a5a9a95327ba8444f3b52de84ae461659fe1ca9f6e3ebd116f7b28f7ff Nov 01 14:36:13.739160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.739130 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7"] Nov 01 14:36:13.996595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:13.996517 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" event={"ID":"0af05ded-ba6c-425b-aa00-6328321ef265","Type":"ContainerStarted","Data":"3ff157a5a9a95327ba8444f3b52de84ae461659fe1ca9f6e3ebd116f7b28f7ff"} Nov 01 14:36:14.309904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.309873 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w"] Nov 01 14:36:14.313986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.313970 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.367594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.367567 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w"] Nov 01 14:36:14.390688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.390656 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.390832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.390724 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.390904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.390824 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgdtr\" (UniqueName: \"kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.491687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.491658 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.491858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.491706 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.491858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.491733 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgdtr\" (UniqueName: \"kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.492334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.492311 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.494101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.494079 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.549111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.549077 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgdtr\" (UniqueName: \"kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr\") pod \"maas-api-revoke-sub-b835e5ed-mgx8w\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.628099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.628069 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:14.995525 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:14.995496 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd493e9fa_b4b4_4c4e_a009_134fe6d8bcbb.slice/crio-b4c3dc3ec208aa38e4ad207355eae64a696bce7c4411b465595815b3b1de07e0 WatchSource:0}: Error finding container b4c3dc3ec208aa38e4ad207355eae64a696bce7c4411b465595815b3b1de07e0: Status 404 returned error can't find the container with id b4c3dc3ec208aa38e4ad207355eae64a696bce7c4411b465595815b3b1de07e0 Nov 01 14:36:14.998123 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:14.998096 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w"] Nov 01 14:36:15.004833 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:15.004811 5473 generic.go:356] "Generic (PLEG): container finished" podID="0af05ded-ba6c-425b-aa00-6328321ef265" containerID="93b847ad2d3c8b03af26eecf41d8bc54b44b8272baebeed3cd3e1652078a9360" exitCode=0 Nov 01 14:36:15.004952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:15.004898 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" event={"ID":"0af05ded-ba6c-425b-aa00-6328321ef265","Type":"ContainerDied","Data":"93b847ad2d3c8b03af26eecf41d8bc54b44b8272baebeed3cd3e1652078a9360"} Nov 01 14:36:15.006267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:15.006246 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" event={"ID":"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb","Type":"ContainerStarted","Data":"b4c3dc3ec208aa38e4ad207355eae64a696bce7c4411b465595815b3b1de07e0"} Nov 01 14:36:16.014106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.014075 5473 generic.go:356] "Generic (PLEG): container finished" podID="d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" containerID="2cd1a3a324341d24cb7a24e1171b8b546a8a97b82883ff429c4276c18eb6a6e8" exitCode=0 Nov 01 14:36:16.014549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.014170 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" event={"ID":"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb","Type":"ContainerDied","Data":"2cd1a3a324341d24cb7a24e1171b8b546a8a97b82883ff429c4276c18eb6a6e8"} Nov 01 14:36:16.162970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.162945 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:16.213350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.213319 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf\" (UniqueName: \"kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf\") pod \"0af05ded-ba6c-425b-aa00-6328321ef265\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " Nov 01 14:36:16.213485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.213475 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca\") pod \"0af05ded-ba6c-425b-aa00-6328321ef265\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " Nov 01 14:36:16.213579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.213566 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca\") pod \"0af05ded-ba6c-425b-aa00-6328321ef265\" (UID: \"0af05ded-ba6c-425b-aa00-6328321ef265\") " Nov 01 14:36:16.213855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.213829 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca" pod "0af05ded-ba6c-425b-aa00-6328321ef265" (UID: "0af05ded-ba6c-425b-aa00-6328321ef265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:36:16.215583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.215556 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf" pod "0af05ded-ba6c-425b-aa00-6328321ef265" (UID: "0af05ded-ba6c-425b-aa00-6328321ef265"). InnerVolumeSpecName "kube-api-access-qtmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:16.215688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.215556 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca" pod "0af05ded-ba6c-425b-aa00-6328321ef265" (UID: "0af05ded-ba6c-425b-aa00-6328321ef265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:16.314926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.314886 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:16.314926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.314920 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af05ded-ba6c-425b-aa00-6328321ef265-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:16.314926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:16.314930 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtmrf\" (UniqueName: \"kubernetes.io/projected/0af05ded-ba6c-425b-aa00-6328321ef265-kube-api-access-qtmrf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:17.022577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.022548 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" Nov 01 14:36:17.023056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.022585 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7" event={"ID":"0af05ded-ba6c-425b-aa00-6328321ef265","Type":"ContainerDied","Data":"3ff157a5a9a95327ba8444f3b52de84ae461659fe1ca9f6e3ebd116f7b28f7ff"} Nov 01 14:36:17.023056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.022646 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff157a5a9a95327ba8444f3b52de84ae461659fe1ca9f6e3ebd116f7b28f7ff" Nov 01 14:36:17.179009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.178982 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:17.223239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.223197 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca\") pod \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " Nov 01 14:36:17.223393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.223268 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr\" (UniqueName: \"kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr\") pod \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " Nov 01 14:36:17.223393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.223317 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca\") pod \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\" (UID: \"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb\") " Nov 01 14:36:17.223694 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.223670 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca" pod "d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" (UID: "d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:36:17.225305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.225283 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca" pod "d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" (UID: "d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:17.225401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.225383 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr" pod "d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" (UID: "d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb"). InnerVolumeSpecName "kube-api-access-xgdtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:17.324700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.324614 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgdtr\" (UniqueName: \"kubernetes.io/projected/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-kube-api-access-xgdtr\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:17.324908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.324892 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:17.325013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.324983 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:17.381174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:17.381138 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:18.034375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:18.034344 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" Nov 01 14:36:18.034862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:18.034377 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w" event={"ID":"d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb","Type":"ContainerDied","Data":"b4c3dc3ec208aa38e4ad207355eae64a696bce7c4411b465595815b3b1de07e0"} Nov 01 14:36:18.034862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:18.034452 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c3dc3ec208aa38e4ad207355eae64a696bce7c4411b465595815b3b1de07e0" Nov 01 14:36:18.125866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:18.125836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:21.838738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:21.838671 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:24.328343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.328302 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" podUID="a9f30019-6fdd-4d97-9271-58fa88ff39c7" containerName="main" containerID="cri-o://fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5" gracePeriod=45 Nov 01 14:36:24.595357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.595297 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:36:24.646055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646027 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home\") pod \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " Nov 01 14:36:24.646245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646091 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm\") pod \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " Nov 01 14:36:24.646245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646121 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9\" (UniqueName: \"kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9\") pod \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " Nov 01 14:36:24.646245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646145 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir\") pod \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " Nov 01 14:36:24.646245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646181 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs\") pod \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " Nov 01 14:36:24.646245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646242 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache\") pod \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\" (UID: \"a9f30019-6fdd-4d97-9271-58fa88ff39c7\") " Nov 01 14:36:24.646537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646332 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home" pod "a9f30019-6fdd-4d97-9271-58fa88ff39c7" (UID: "a9f30019-6fdd-4d97-9271-58fa88ff39c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:36:24.646537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646483 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir" pod "a9f30019-6fdd-4d97-9271-58fa88ff39c7" (UID: "a9f30019-6fdd-4d97-9271-58fa88ff39c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:36:24.646537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646497 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache" pod "a9f30019-6fdd-4d97-9271-58fa88ff39c7" (UID: "a9f30019-6fdd-4d97-9271-58fa88ff39c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:36:24.646886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646865 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:24.646936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646893 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:24.646936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.646911 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:24.648481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.648452 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs" pod "a9f30019-6fdd-4d97-9271-58fa88ff39c7" (UID: "a9f30019-6fdd-4d97-9271-58fa88ff39c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:24.648481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.648473 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm" pod "a9f30019-6fdd-4d97-9271-58fa88ff39c7" (UID: "a9f30019-6fdd-4d97-9271-58fa88ff39c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:36:24.648623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.648492 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9" pod "a9f30019-6fdd-4d97-9271-58fa88ff39c7" (UID: "a9f30019-6fdd-4d97-9271-58fa88ff39c7"). InnerVolumeSpecName "kube-api-access-rhcc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:24.747917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.747886 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9f30019-6fdd-4d97-9271-58fa88ff39c7-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:24.747917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.747913 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9f30019-6fdd-4d97-9271-58fa88ff39c7-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:24.747917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:24.747923 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhcc9\" (UniqueName: \"kubernetes.io/projected/a9f30019-6fdd-4d97-9271-58fa88ff39c7-kube-api-access-rhcc9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:25.004135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004060 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g"] Nov 01 14:36:25.004614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004601 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" containerName="revoke-keys" Nov 01 14:36:25.004671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004615 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" containerName="revoke-keys" Nov 01 14:36:25.004671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004626 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a9f30019-6fdd-4d97-9271-58fa88ff39c7" containerName="main" Nov 01 14:36:25.004671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004632 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f30019-6fdd-4d97-9271-58fa88ff39c7" containerName="main" Nov 01 14:36:25.004671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004661 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0af05ded-ba6c-425b-aa00-6328321ef265" containerName="revoke-keys" Nov 01 14:36:25.004671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004666 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af05ded-ba6c-425b-aa00-6328321ef265" containerName="revoke-keys" Nov 01 14:36:25.004813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004747 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" containerName="revoke-keys" Nov 01 14:36:25.004813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004755 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f30019-6fdd-4d97-9271-58fa88ff39c7" containerName="main" Nov 01 14:36:25.004813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.004763 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af05ded-ba6c-425b-aa00-6328321ef265" containerName="revoke-keys" Nov 01 14:36:25.008264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.008246 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.051316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.051283 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.051468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.051428 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.051468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.051445 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flwcx\" (UniqueName: \"kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.060655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.060626 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:36:25.060788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.060723 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:36:25.062210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.062186 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g"] Nov 01 14:36:25.121285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.121256 5473 generic.go:356] "Generic (PLEG): container finished" podID="a9f30019-6fdd-4d97-9271-58fa88ff39c7" containerID="fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5" exitCode=0 Nov 01 14:36:25.121402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.121311 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" Nov 01 14:36:25.121402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.121339 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" event={"ID":"a9f30019-6fdd-4d97-9271-58fa88ff39c7","Type":"ContainerDied","Data":"fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5"} Nov 01 14:36:25.121402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.121381 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd" event={"ID":"a9f30019-6fdd-4d97-9271-58fa88ff39c7","Type":"ContainerDied","Data":"8d3ce25ecf3e37270e71662d74d8586a95d40ee61345bb1f62f124f8918ec9a4"} Nov 01 14:36:25.121508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.121403 5473 scope.go:122] "RemoveContainer" containerID="fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5" Nov 01 14:36:25.131194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.131177 5473 scope.go:122] "RemoveContainer" containerID="fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5" Nov 01 14:36:25.131508 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:36:25.131483 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5\": container with ID starting with fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5 not found: ID does not exist" containerID="fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5" Nov 01 14:36:25.131622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.131512 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5"} err="failed to get container status \"fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5\": rpc error: code = NotFound desc = could not find container \"fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5\": container with ID starting with fb32e8c2b89654d29097290ae2184a9394f029da471c2cdacc309d82561264c5 not found: ID does not exist" Nov 01 14:36:25.152723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.152699 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.152832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.152727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flwcx\" (UniqueName: \"kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.152832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.152802 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.153433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.153415 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.154881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.154859 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.209830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.209804 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flwcx\" (UniqueName: \"kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx\") pod \"maas-api-revoke-sub-602a3a60-5m67g\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.290115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.290089 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd"] Nov 01 14:36:25.294344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.294324 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/e2e-conflict-b-a12d4d90-kserve-679d75bfd-8g6qd"] Nov 01 14:36:25.322322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.322304 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:25.693544 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:25.693514 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5268ed_3ff3_44c1_a1ba_bee37afbcc68.slice/crio-31e59670921f13818e3428a93cd5a248bdfb7cb84dd0db931fa0ac9272104a7f WatchSource:0}: Error finding container 31e59670921f13818e3428a93cd5a248bdfb7cb84dd0db931fa0ac9272104a7f: Status 404 returned error can't find the container with id 31e59670921f13818e3428a93cd5a248bdfb7cb84dd0db931fa0ac9272104a7f Nov 01 14:36:25.697499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:25.697477 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g"] Nov 01 14:36:26.132424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:26.132389 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" event={"ID":"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68","Type":"ContainerStarted","Data":"1afa9db20cfbdc40fa8efe9e1b5450119e492c2d0db04828f8983e4b9543c87b"} Nov 01 14:36:26.132609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:26.132441 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" event={"ID":"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68","Type":"ContainerStarted","Data":"31e59670921f13818e3428a93cd5a248bdfb7cb84dd0db931fa0ac9272104a7f"} Nov 01 14:36:26.250180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:26.250068 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" podStartSLOduration=2.250052387 podStartE2EDuration="2.250052387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:26.248900377 +0000 UTC m=+1125.307052227" watchObservedRunningTime="2026-11-01 14:36:26.250052387 +0000 UTC m=+1125.308204239" Nov 01 14:36:26.610330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:26.610306 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:27.081931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:27.081898 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f30019-6fdd-4d97-9271-58fa88ff39c7" path="/var/lib/kubelet/pods/a9f30019-6fdd-4d97-9271-58fa88ff39c7/volumes" Nov 01 14:36:27.141530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:27.141494 5473 generic.go:356] "Generic (PLEG): container finished" podID="3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" containerID="1afa9db20cfbdc40fa8efe9e1b5450119e492c2d0db04828f8983e4b9543c87b" exitCode=0 Nov 01 14:36:27.141690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:27.141539 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" event={"ID":"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68","Type":"ContainerDied","Data":"1afa9db20cfbdc40fa8efe9e1b5450119e492c2d0db04828f8983e4b9543c87b"} Nov 01 14:36:28.296277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.296251 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:28.391364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.391332 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca\") pod \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " Nov 01 14:36:28.391531 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.391501 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca\") pod \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " Nov 01 14:36:28.391665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.391647 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx\" (UniqueName: \"kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx\") pod \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\" (UID: \"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68\") " Nov 01 14:36:28.391758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.391740 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca" pod "3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" (UID: "3f5268ed-3ff3-44c1-a1ba-bee37afbcc68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:36:28.392016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.391995 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:28.393516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.393491 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca" pod "3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" (UID: "3f5268ed-3ff3-44c1-a1ba-bee37afbcc68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:28.393637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.393617 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx" pod "3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" (UID: "3f5268ed-3ff3-44c1-a1ba-bee37afbcc68"). InnerVolumeSpecName "kube-api-access-flwcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:28.493070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.493034 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flwcx\" (UniqueName: \"kubernetes.io/projected/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-kube-api-access-flwcx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:28.493070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.493067 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:28.807752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.807721 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v"] Nov 01 14:36:28.808313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.808298 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" containerName="revoke-keys" Nov 01 14:36:28.808386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.808313 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" containerName="revoke-keys" Nov 01 14:36:28.808427 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.808417 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" containerName="revoke-keys" Nov 01 14:36:28.818557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.812851 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.863747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.863712 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v"] Nov 01 14:36:28.897351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.897319 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.897554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.897491 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.897693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.897672 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkh5\" (UniqueName: \"kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.999030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.999001 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkh5\" (UniqueName: \"kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.999241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.999042 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.999241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.999121 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:28.999862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:28.999838 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:29.001521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.001494 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:29.058843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.058776 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkh5\" (UniqueName: \"kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5\") pod \"maas-api-revoke-sub-875a071b-mjq9v\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:29.134177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.134144 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:29.159418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.159390 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" Nov 01 14:36:29.159536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.159423 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g" event={"ID":"3f5268ed-3ff3-44c1-a1ba-bee37afbcc68","Type":"ContainerDied","Data":"31e59670921f13818e3428a93cd5a248bdfb7cb84dd0db931fa0ac9272104a7f"} Nov 01 14:36:29.159536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.159479 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31e59670921f13818e3428a93cd5a248bdfb7cb84dd0db931fa0ac9272104a7f" Nov 01 14:36:29.501491 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:29.501460 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73567e3_6637_41b5_99c9_f78e60ea154f.slice/crio-a52a83a2f9c6484fad12cecf96b9027b7ef23ad0eb16ad97ee9469042195191d WatchSource:0}: Error finding container a52a83a2f9c6484fad12cecf96b9027b7ef23ad0eb16ad97ee9469042195191d: Status 404 returned error can't find the container with id a52a83a2f9c6484fad12cecf96b9027b7ef23ad0eb16ad97ee9469042195191d Nov 01 14:36:29.506854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:29.506831 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v"] Nov 01 14:36:30.169068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:30.168991 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" event={"ID":"e73567e3-6637-41b5-99c9-f78e60ea154f","Type":"ContainerStarted","Data":"84ac1366631eb620ae8af09a4802354eb8c3f70fda8eb26c8dc7e963a39c076e"} Nov 01 14:36:30.169068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:30.169038 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" event={"ID":"e73567e3-6637-41b5-99c9-f78e60ea154f","Type":"ContainerStarted","Data":"a52a83a2f9c6484fad12cecf96b9027b7ef23ad0eb16ad97ee9469042195191d"} Nov 01 14:36:30.286197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:30.286151 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" podStartSLOduration=2.286137201 podStartE2EDuration="2.286137201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:30.283161512 +0000 UTC m=+1129.341313362" watchObservedRunningTime="2026-11-01 14:36:30.286137201 +0000 UTC m=+1129.344289052" Nov 01 14:36:31.068818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:31.068788 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:31.108961 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:36:31.108931 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:36:31.176517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:31.176489 5473 generic.go:356] "Generic (PLEG): container finished" podID="e73567e3-6637-41b5-99c9-f78e60ea154f" containerID="84ac1366631eb620ae8af09a4802354eb8c3f70fda8eb26c8dc7e963a39c076e" exitCode=0 Nov 01 14:36:31.176704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:31.176574 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" event={"ID":"e73567e3-6637-41b5-99c9-f78e60ea154f","Type":"ContainerDied","Data":"84ac1366631eb620ae8af09a4802354eb8c3f70fda8eb26c8dc7e963a39c076e"} Nov 01 14:36:32.316744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.316720 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:32.440143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.440059 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca\") pod \"e73567e3-6637-41b5-99c9-f78e60ea154f\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " Nov 01 14:36:32.440143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.440131 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5\" (UniqueName: \"kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5\") pod \"e73567e3-6637-41b5-99c9-f78e60ea154f\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " Nov 01 14:36:32.440402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.440259 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca\") pod \"e73567e3-6637-41b5-99c9-f78e60ea154f\" (UID: \"e73567e3-6637-41b5-99c9-f78e60ea154f\") " Nov 01 14:36:32.440669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.440639 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca" pod "e73567e3-6637-41b5-99c9-f78e60ea154f" (UID: "e73567e3-6637-41b5-99c9-f78e60ea154f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:36:32.441193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.441170 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:32.442101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.442076 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca" pod "e73567e3-6637-41b5-99c9-f78e60ea154f" (UID: "e73567e3-6637-41b5-99c9-f78e60ea154f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:32.442465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.442450 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5" pod "e73567e3-6637-41b5-99c9-f78e60ea154f" (UID: "e73567e3-6637-41b5-99c9-f78e60ea154f"). InnerVolumeSpecName "kube-api-access-mpkh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:32.541832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.541802 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e73567e3-6637-41b5-99c9-f78e60ea154f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:32.541832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:32.541827 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkh5\" (UniqueName: \"kubernetes.io/projected/e73567e3-6637-41b5-99c9-f78e60ea154f-kube-api-access-mpkh5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:33.208611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:33.208565 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" event={"ID":"e73567e3-6637-41b5-99c9-f78e60ea154f","Type":"ContainerDied","Data":"a52a83a2f9c6484fad12cecf96b9027b7ef23ad0eb16ad97ee9469042195191d"} Nov 01 14:36:33.208813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:33.208640 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a52a83a2f9c6484fad12cecf96b9027b7ef23ad0eb16ad97ee9469042195191d" Nov 01 14:36:33.208813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:33.208722 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v" Nov 01 14:36:35.871778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:35.871743 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:35.938279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:35.938242 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6"] Nov 01 14:36:35.938824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:35.938808 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e73567e3-6637-41b5-99c9-f78e60ea154f" containerName="revoke-keys" Nov 01 14:36:35.938891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:35.938826 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73567e3-6637-41b5-99c9-f78e60ea154f" containerName="revoke-keys" Nov 01 14:36:35.938947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:35.938936 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73567e3-6637-41b5-99c9-f78e60ea154f" containerName="revoke-keys" Nov 01 14:36:35.942653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:35.942630 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.035092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.035056 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6"] Nov 01 14:36:36.046311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.046264 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqm8r\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.046484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.046345 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.046484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.046439 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.046641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.046486 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.046641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.046582 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.046724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.046680 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.047257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.047206 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.047312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.047286 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.047347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.047317 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148576 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148612 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148630 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148655 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqm8r\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148682 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148708 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148729 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148749 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.148991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.148786 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.149290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.149025 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.149290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.149139 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.149290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.149189 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.149449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.149378 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.149701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.149678 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.150967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.150947 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.151262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.151238 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.206657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.206629 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqm8r\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.207228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.207202 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.259031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.259005 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:36.679141 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:36.679025 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007c0d84_c42e_4d3b_ad3c_cef5e43bf883.slice/crio-739a568f44cea6dbc8a70c3bc81bb871203fcfa9893b912c762b9e723de082eb WatchSource:0}: Error finding container 739a568f44cea6dbc8a70c3bc81bb871203fcfa9893b912c762b9e723de082eb: Status 404 returned error can't find the container with id 739a568f44cea6dbc8a70c3bc81bb871203fcfa9893b912c762b9e723de082eb Nov 01 14:36:36.682296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.682210 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:36.682377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.682348 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:36.682433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.682391 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:36.683774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:36.683751 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6"] Nov 01 14:36:37.258419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:37.258375 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" event={"ID":"007c0d84-c42e-4d3b-ad3c-cef5e43bf883","Type":"ContainerStarted","Data":"5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c"} Nov 01 14:36:37.258910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:37.258438 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" event={"ID":"007c0d84-c42e-4d3b-ad3c-cef5e43bf883","Type":"ContainerStarted","Data":"739a568f44cea6dbc8a70c3bc81bb871203fcfa9893b912c762b9e723de082eb"} Nov 01 14:36:37.259693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:37.259677 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:37.261276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:37.261253 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:37.261399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:37.261298 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:37.379556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:37.379500 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podStartSLOduration=2.37948306 podStartE2EDuration="2.37948306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:37.376538612 +0000 UTC m=+1136.434690461" watchObservedRunningTime="2026-11-01 14:36:37.37948306 +0000 UTC m=+1136.437634911" Nov 01 14:36:38.260302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:38.260265 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:38.260723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:38.260319 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:39.259721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:39.259685 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:39.259904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:39.259738 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:40.259972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:40.259930 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:40.260368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:40.259988 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:40.439241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:40.434127 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:41.199069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.199040 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll"] Nov 01 14:36:41.203285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.203263 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.256155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.256126 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll"] Nov 01 14:36:41.259452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.259423 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:41.259608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.259470 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:41.301714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.301688 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:36:41.301714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.301714 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:36:41.315451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.315426 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.315675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.315655 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss7h5\" (UniqueName: \"kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.315792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.315771 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.416674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.416647 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss7h5\" (UniqueName: \"kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.416837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.416720 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.416837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.416769 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.417453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.417432 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.419029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.419010 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.487080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.487013 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss7h5\" (UniqueName: \"kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5\") pod \"maas-api-revoke-sub-068d5f54-7lmll\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.517018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.516991 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:41.912709 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:41.912678 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b0ff2e_4cbb_4f91_a765_3d52ed71c654.slice/crio-56e13b24a13b1e9a7d55c3222e033b32af0389fba3d5079ac821d2ce28b6355e WatchSource:0}: Error finding container 56e13b24a13b1e9a7d55c3222e033b32af0389fba3d5079ac821d2ce28b6355e: Status 404 returned error can't find the container with id 56e13b24a13b1e9a7d55c3222e033b32af0389fba3d5079ac821d2ce28b6355e Nov 01 14:36:41.916973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.916949 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll"] Nov 01 14:36:41.999001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:41.998973 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898"] Nov 01 14:36:42.004100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.004082 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.014513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.014489 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc"] Nov 01 14:36:42.019042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.019025 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.061014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.060986 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898"] Nov 01 14:36:42.108715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.108685 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc"] Nov 01 14:36:42.124155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124120 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124255 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124309 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124336 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.124471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124367 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124420 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124445 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vphp\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.124690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124470 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124500 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.124690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124530 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124578 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.124690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124616 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.124690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124666 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.125001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124694 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.125001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124745 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.125001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124779 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.125001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124825 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.125001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.124888 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qff\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.248239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.237068 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qff\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.248239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.237120 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.249749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.249693 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.237229 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251329 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251359 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.251520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251396 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251457 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251482 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vphp\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.251520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251516 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251548 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.251715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251590 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251644 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.251715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251684 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.251715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251690 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.252040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251766 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.252040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251796 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.252040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251871 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.252040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251922 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.252040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.251993 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.252040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.252003 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.252400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.252242 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.253641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.252516 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.253641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.253077 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.253641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.253174 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.253641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.253182 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.253641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.253503 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.253641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.253549 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.254014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.253690 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.255506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.255480 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.255610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.255581 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.255809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.255790 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.259805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.259774 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:42.259901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.259823 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:42.292137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.292111 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" event={"ID":"45b0ff2e-4cbb-4f91-a765-3d52ed71c654","Type":"ContainerStarted","Data":"56e13b24a13b1e9a7d55c3222e033b32af0389fba3d5079ac821d2ce28b6355e"} Nov 01 14:36:42.306223 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.306194 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qff\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.309895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.309874 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-mr898\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.310063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.310042 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.310371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.310353 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vphp\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.321127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.321110 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:42.351059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.351028 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:42.820097 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:42.820067 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc2ec48_c811_4935_8a44_32edbcd9eeb6.slice/crio-ccb2c4333bc8714014606b75c8fb62f94f0be28f4fecb2b140ab87f24c02760c WatchSource:0}: Error finding container ccb2c4333bc8714014606b75c8fb62f94f0be28f4fecb2b140ab87f24c02760c: Status 404 returned error can't find the container with id ccb2c4333bc8714014606b75c8fb62f94f0be28f4fecb2b140ab87f24c02760c Nov 01 14:36:42.821467 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:42.821440 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf1df67_e60a_4173_a438_2931bd477cd3.slice/crio-b1fa10399a304d4c1cd38a96b40ba0b950d397703eee660f5430b57c6424d966 WatchSource:0}: Error finding container b1fa10399a304d4c1cd38a96b40ba0b950d397703eee660f5430b57c6424d966: Status 404 returned error can't find the container with id b1fa10399a304d4c1cd38a96b40ba0b950d397703eee660f5430b57c6424d966 Nov 01 14:36:42.822932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.822898 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:42.823045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.822982 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:42.823045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.823011 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:42.824354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.824329 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898"] Nov 01 14:36:42.825442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.825412 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:42.825549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.825476 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:42.825549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.825517 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:42.828370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:42.828345 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc"] Nov 01 14:36:43.259778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.259683 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:36:43.259778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.259737 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:36:43.304202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.304158 5473 generic.go:356] "Generic (PLEG): container finished" podID="45b0ff2e-4cbb-4f91-a765-3d52ed71c654" containerID="5c6643e54d69e81bbac4a28bd7271615b4a8e60c58562695ca470f40ab39f1dc" exitCode=0 Nov 01 14:36:43.304395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.304251 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" event={"ID":"45b0ff2e-4cbb-4f91-a765-3d52ed71c654","Type":"ContainerDied","Data":"5c6643e54d69e81bbac4a28bd7271615b4a8e60c58562695ca470f40ab39f1dc"} Nov 01 14:36:43.306195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.306164 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" event={"ID":"bbf1df67-e60a-4173-a438-2931bd477cd3","Type":"ContainerStarted","Data":"fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef"} Nov 01 14:36:43.306335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.306233 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" event={"ID":"bbf1df67-e60a-4173-a438-2931bd477cd3","Type":"ContainerStarted","Data":"b1fa10399a304d4c1cd38a96b40ba0b950d397703eee660f5430b57c6424d966"} Nov 01 14:36:43.307980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.307947 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" event={"ID":"9bc2ec48-c811-4935-8a44-32edbcd9eeb6","Type":"ContainerStarted","Data":"93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d"} Nov 01 14:36:43.308090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.307996 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" event={"ID":"9bc2ec48-c811-4935-8a44-32edbcd9eeb6","Type":"ContainerStarted","Data":"ccb2c4333bc8714014606b75c8fb62f94f0be28f4fecb2b140ab87f24c02760c"} Nov 01 14:36:43.321764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.321736 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:43.323248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.323211 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:43.323376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.323276 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:43.351460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.351418 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:43.353504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.353476 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:43.353640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.353536 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:43.717635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:43.717566 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podStartSLOduration=2.7175461629999997 podStartE2EDuration="2.717546163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:43.569582115 +0000 UTC m=+1142.627733947" watchObservedRunningTime="2026-11-01 14:36:43.717546163 +0000 UTC m=+1142.775698013" Nov 01 14:36:44.278345 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.278311 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:44.278934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.278912 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:44.279693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.279669 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:36:44.321964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.321942 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:44.322241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.321989 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:44.374284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.364717 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:44.374284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.364778 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:44.406122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.406064 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podStartSLOduration=3.406044899 podStartE2EDuration="3.406044899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:43.718142364 +0000 UTC m=+1142.776294214" watchObservedRunningTime="2026-11-01 14:36:44.406044899 +0000 UTC m=+1143.464196750" Nov 01 14:36:44.517821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.517789 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn"] Nov 01 14:36:44.522063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.522043 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.611364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.611332 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn"] Nov 01 14:36:44.690774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.690745 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:44.701383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.701356 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.701547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.701500 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczqv\" (UniqueName: \"kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.701612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.701556 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.802821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.802745 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca\") pod \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " Nov 01 14:36:44.802981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.802878 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5\" (UniqueName: \"kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5\") pod \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " Nov 01 14:36:44.802981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.802946 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca\") pod \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\" (UID: \"45b0ff2e-4cbb-4f91-a765-3d52ed71c654\") " Nov 01 14:36:44.803170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.803152 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.803410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.803386 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczqv\" (UniqueName: \"kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.803517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.803496 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.803992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.803948 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca" pod "45b0ff2e-4cbb-4f91-a765-3d52ed71c654" (UID: "45b0ff2e-4cbb-4f91-a765-3d52ed71c654"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:36:44.805211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.805183 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.805469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.805443 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5" pod "45b0ff2e-4cbb-4f91-a765-3d52ed71c654" (UID: "45b0ff2e-4cbb-4f91-a765-3d52ed71c654"). InnerVolumeSpecName "kube-api-access-ss7h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:44.805997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.805971 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca" pod "45b0ff2e-4cbb-4f91-a765-3d52ed71c654" (UID: "45b0ff2e-4cbb-4f91-a765-3d52ed71c654"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:44.807060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.807037 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.888403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.888323 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczqv\" (UniqueName: \"kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv\") pod \"maas-api-revoke-sub-6b8a474d-q2kjn\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:44.902760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.902722 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:44.905187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.904958 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:44.905187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.904993 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss7h5\" (UniqueName: \"kubernetes.io/projected/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-kube-api-access-ss7h5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:44.905187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.905008 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45b0ff2e-4cbb-4f91-a765-3d52ed71c654-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:44.995618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:44.986696 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:45.321723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.321691 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:45.321869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.321750 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:45.340241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.336464 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" Nov 01 14:36:45.364235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.351789 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll" event={"ID":"45b0ff2e-4cbb-4f91-a765-3d52ed71c654","Type":"ContainerDied","Data":"56e13b24a13b1e9a7d55c3222e033b32af0389fba3d5079ac821d2ce28b6355e"} Nov 01 14:36:45.364235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.351847 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e13b24a13b1e9a7d55c3222e033b32af0389fba3d5079ac821d2ce28b6355e" Nov 01 14:36:45.377362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.374409 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:45.377362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.374491 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:45.479195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:45.479168 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn"] Nov 01 14:36:46.321932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.321902 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:46.322090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.321956 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:46.362251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.360311 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:46.362251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.360372 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:46.379291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.378561 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" event={"ID":"4ac984f1-d55b-49b3-ad34-ef434086d7b5","Type":"ContainerStarted","Data":"1d0f0cee62eb32f7045ec78704de2bb292ec261fe17a3cc02e2fa5f2897b4f11"} Nov 01 14:36:46.379291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.378620 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" event={"ID":"4ac984f1-d55b-49b3-ad34-ef434086d7b5","Type":"ContainerStarted","Data":"83885051913bc6db9f32cc27b79ac308f333c385ad2fcb6824906953c25ebdc6"} Nov 01 14:36:46.499355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:46.499305 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" podStartSLOduration=2.499289985 podStartE2EDuration="2.499289985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:46.494733505 +0000 UTC m=+1145.552885356" watchObservedRunningTime="2026-11-01 14:36:46.499289985 +0000 UTC m=+1145.557441834" Nov 01 14:36:47.321919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:47.321885 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:47.322093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:47.321947 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:47.356240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:47.352554 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:47.356240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:47.352614 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:47.388892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:47.388860 5473 generic.go:356] "Generic (PLEG): container finished" podID="4ac984f1-d55b-49b3-ad34-ef434086d7b5" containerID="1d0f0cee62eb32f7045ec78704de2bb292ec261fe17a3cc02e2fa5f2897b4f11" exitCode=0 Nov 01 14:36:47.389270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:47.389015 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" event={"ID":"4ac984f1-d55b-49b3-ad34-ef434086d7b5","Type":"ContainerDied","Data":"1d0f0cee62eb32f7045ec78704de2bb292ec261fe17a3cc02e2fa5f2897b4f11"} Nov 01 14:36:48.321692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:48.321659 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:48.321848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:48.321714 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:48.351622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:48.351586 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:48.351878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:48.351850 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:48.830679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:48.830645 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:49.314239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.313315 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:49.355241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.351733 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:49.379394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.379294 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:49.405240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.368017 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:49.405506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.405473 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:49.507274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.441821 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca\") pod \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " Nov 01 14:36:49.507274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.441966 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv\" (UniqueName: \"kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv\") pod \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " Nov 01 14:36:49.507274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.442065 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca\") pod \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\" (UID: \"4ac984f1-d55b-49b3-ad34-ef434086d7b5\") " Nov 01 14:36:49.507274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.451134 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca" pod "4ac984f1-d55b-49b3-ad34-ef434086d7b5" (UID: "4ac984f1-d55b-49b3-ad34-ef434086d7b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:36:49.556239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.548617 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:49.589176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.569274 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca" pod "4ac984f1-d55b-49b3-ad34-ef434086d7b5" (UID: "4ac984f1-d55b-49b3-ad34-ef434086d7b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:36:49.627879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.627844 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv" pod "4ac984f1-d55b-49b3-ad34-ef434086d7b5" (UID: "4ac984f1-d55b-49b3-ad34-ef434086d7b5"). InnerVolumeSpecName "kube-api-access-vczqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:36:49.638293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.633695 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" event={"ID":"4ac984f1-d55b-49b3-ad34-ef434086d7b5","Type":"ContainerDied","Data":"83885051913bc6db9f32cc27b79ac308f333c385ad2fcb6824906953c25ebdc6"} Nov 01 14:36:49.638293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.633750 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83885051913bc6db9f32cc27b79ac308f333c385ad2fcb6824906953c25ebdc6" Nov 01 14:36:49.638293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.633820 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn" Nov 01 14:36:49.670543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.661187 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac984f1-d55b-49b3-ad34-ef434086d7b5-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:49.670543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.661243 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vczqv\" (UniqueName: \"kubernetes.io/projected/4ac984f1-d55b-49b3-ad34-ef434086d7b5-kube-api-access-vczqv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:36:49.867241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:49.849479 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:50.353501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:50.353463 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Nov 01 14:36:50.353780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:50.353748 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Nov 01 14:36:50.377425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:50.377343 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 01 14:36:50.377650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:50.377620 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 01 14:36:51.127871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.127245 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c"] Nov 01 14:36:51.128407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.128070 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95ff8b3-dzs6c"] Nov 01 14:36:51.134090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.134062 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9"] Nov 01 14:36:51.135110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.135078 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4ac984f1-d55b-49b3-ad34-ef434086d7b5" containerName="revoke-keys" Nov 01 14:36:51.135110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.135108 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac984f1-d55b-49b3-ad34-ef434086d7b5" containerName="revoke-keys" Nov 01 14:36:51.135356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.135145 5473 cpu_manager.go:414] "containerMap: removing container" podUID="45b0ff2e-4cbb-4f91-a765-3d52ed71c654" containerName="revoke-keys" Nov 01 14:36:51.135356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.135154 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b0ff2e-4cbb-4f91-a765-3d52ed71c654" containerName="revoke-keys" Nov 01 14:36:51.135356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.135339 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b0ff2e-4cbb-4f91-a765-3d52ed71c654" containerName="revoke-keys" Nov 01 14:36:51.135524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.135361 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac984f1-d55b-49b3-ad34-ef434086d7b5" containerName="revoke-keys" Nov 01 14:36:51.139504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.139480 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.215828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.215801 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9"] Nov 01 14:36:51.288595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.288557 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.288758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.288643 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.288847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.288777 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.288944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.288923 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.289027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.288976 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.289087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.289071 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.289162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.289144 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.289286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.289268 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.289739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.289709 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8747\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391352 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391430 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8747\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391481 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391540 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391581 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391672 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391697 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391767 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.391908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.391814 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.392284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.392256 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.392562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.392539 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.392645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.392598 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.392828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.392806 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.393498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.393470 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.394643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.394615 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.395230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.395191 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.449610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.449570 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.449858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.449838 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8747\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.456962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.456942 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:51.891350 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:51.891319 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff736db_b409_4a27_b198_0108ac29adc5.slice/crio-e335492e781adb87219eb8d3db8ca304293aa3effffee0b493c5b9ff53c26111 WatchSource:0}: Error finding container e335492e781adb87219eb8d3db8ca304293aa3effffee0b493c5b9ff53c26111: Status 404 returned error can't find the container with id e335492e781adb87219eb8d3db8ca304293aa3effffee0b493c5b9ff53c26111 Nov 01 14:36:51.893566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.893544 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9"] Nov 01 14:36:51.894496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.894454 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:51.894563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.894541 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:51.894598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:51.894585 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:36:52.364240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.363369 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:36:52.364240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.363446 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:36:52.394238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.393431 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:36:52.394238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.393523 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:36:52.682640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.675656 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" event={"ID":"0ff736db-b409-4a27-b198-0108ac29adc5","Type":"ContainerStarted","Data":"967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0"} Nov 01 14:36:52.684878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.684843 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" event={"ID":"0ff736db-b409-4a27-b198-0108ac29adc5","Type":"ContainerStarted","Data":"e335492e781adb87219eb8d3db8ca304293aa3effffee0b493c5b9ff53c26111"} Nov 01 14:36:52.781756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.777303 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:52.781756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.778513 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:52.793968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.793874 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podStartSLOduration=1.7938607100000001 podStartE2EDuration="1.79386071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:52.792327255 +0000 UTC m=+1151.850479106" watchObservedRunningTime="2026-11-01 14:36:52.79386071 +0000 UTC m=+1151.852012560" Nov 01 14:36:52.828557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.828526 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:36:52.912681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.912645 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:52.913108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.913082 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:52.913849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:52.913828 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:36:53.083775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:53.083746 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69" path="/var/lib/kubelet/pods/74ee02a8-2c65-4e05-9cc8-5aa3d56f2b69/volumes" Nov 01 14:36:53.458101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:53.458015 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:53.459862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:53.459837 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 14:36:53.459984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:53.459895 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 14:36:54.457355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:54.457318 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 14:36:54.457518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:54.457375 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 14:36:54.747288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:54.744476 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:55.457341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:55.457305 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 14:36:55.457541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:55.457377 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 14:36:56.458209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:56.458176 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 14:36:56.458663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:56.458242 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 14:36:57.214267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.214234 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh"] Nov 01 14:36:57.218258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.218242 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.269841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.269806 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh"] Nov 01 14:36:57.284959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.284931 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.285086 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.284986 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dj2\" (UniqueName: \"kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.285283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.285255 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.315918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.315892 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:36:57.316052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.315939 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:36:57.386241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.386175 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dj2\" (UniqueName: \"kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.386423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.386318 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.386423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.386392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.386986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.386961 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.388652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.388634 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.447236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.446871 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dj2\" (UniqueName: \"kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2\") pod \"maas-api-revoke-sub-b771e40a-7bkjh\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.457665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.457636 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 14:36:57.457807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.457691 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 14:36:57.557226 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.557190 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:36:57.953285 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:36:57.953203 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d76c85_4993_4586_b518_0be7e1a49232.slice/crio-f09e3db564f561af564b1fc8e9c2c5d27a6d98bbe6dd9367dd499128cbb83d00 WatchSource:0}: Error finding container f09e3db564f561af564b1fc8e9c2c5d27a6d98bbe6dd9367dd499128cbb83d00: Status 404 returned error can't find the container with id f09e3db564f561af564b1fc8e9c2c5d27a6d98bbe6dd9367dd499128cbb83d00 Nov 01 14:36:57.954010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:57.953788 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh"] Nov 01 14:36:58.457967 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:58.457931 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 01 14:36:58.458129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:58.457987 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 01 14:36:58.719512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:58.719421 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" event={"ID":"22d76c85-4993-4586-b518-0be7e1a49232","Type":"ContainerStarted","Data":"319e23f5ddefbe628222e16b54dab8364de80fd83f4eda0236d17ea89685c676"} Nov 01 14:36:58.719512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:58.719476 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" event={"ID":"22d76c85-4993-4586-b518-0be7e1a49232","Type":"ContainerStarted","Data":"f09e3db564f561af564b1fc8e9c2c5d27a6d98bbe6dd9367dd499128cbb83d00"} Nov 01 14:36:58.832998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:58.832947 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" podStartSLOduration=1.832932221 podStartE2EDuration="1.832932221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:36:58.832229982 +0000 UTC m=+1157.890381829" watchObservedRunningTime="2026-11-01 14:36:58.832932221 +0000 UTC m=+1157.891084071" Nov 01 14:36:59.115535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:59.115506 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:36:59.588230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:59.588188 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:59.589423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:59.589398 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:59.589930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:59.589909 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:36:59.730521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:59.730494 5473 generic.go:356] "Generic (PLEG): container finished" podID="22d76c85-4993-4586-b518-0be7e1a49232" containerID="319e23f5ddefbe628222e16b54dab8364de80fd83f4eda0236d17ea89685c676" exitCode=0 Nov 01 14:36:59.730909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:36:59.730582 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" event={"ID":"22d76c85-4993-4586-b518-0be7e1a49232","Type":"ContainerDied","Data":"319e23f5ddefbe628222e16b54dab8364de80fd83f4eda0236d17ea89685c676"} Nov 01 14:37:00.884085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.884059 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:37:00.928412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.928391 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca\") pod \"22d76c85-4993-4586-b518-0be7e1a49232\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " Nov 01 14:37:00.928557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.928421 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca\") pod \"22d76c85-4993-4586-b518-0be7e1a49232\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " Nov 01 14:37:00.928557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.928528 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2\" (UniqueName: \"kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2\") pod \"22d76c85-4993-4586-b518-0be7e1a49232\" (UID: \"22d76c85-4993-4586-b518-0be7e1a49232\") " Nov 01 14:37:00.928771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.928746 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca" pod "22d76c85-4993-4586-b518-0be7e1a49232" (UID: "22d76c85-4993-4586-b518-0be7e1a49232"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:00.928971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.928955 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:00.930472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.930452 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca" pod "22d76c85-4993-4586-b518-0be7e1a49232" (UID: "22d76c85-4993-4586-b518-0be7e1a49232"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:00.930550 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:00.930532 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2" pod "22d76c85-4993-4586-b518-0be7e1a49232" (UID: "22d76c85-4993-4586-b518-0be7e1a49232"). InnerVolumeSpecName "kube-api-access-h4dj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:01.030455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.030420 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22d76c85-4993-4586-b518-0be7e1a49232-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:01.030455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.030445 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dj2\" (UniqueName: \"kubernetes.io/projected/22d76c85-4993-4586-b518-0be7e1a49232-kube-api-access-h4dj2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:01.047587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.047564 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b"] Nov 01 14:37:01.048185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.048167 5473 cpu_manager.go:414] "containerMap: removing container" podUID="22d76c85-4993-4586-b518-0be7e1a49232" containerName="revoke-keys" Nov 01 14:37:01.048241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.048187 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d76c85-4993-4586-b518-0be7e1a49232" containerName="revoke-keys" Nov 01 14:37:01.048324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.048311 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d76c85-4993-4586-b518-0be7e1a49232" containerName="revoke-keys" Nov 01 14:37:01.051918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.051900 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.104835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.104803 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b"] Nov 01 14:37:01.119573 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:37:01.119537 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:37:01.131946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.131760 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.132537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.132148 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.132537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.132255 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjr4\" (UniqueName: \"kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.233857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.233828 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.234007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.233869 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjr4\" (UniqueName: \"kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.234007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.233931 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.234481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.234458 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.236276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.236259 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.290625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.290574 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjr4\" (UniqueName: \"kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4\") pod \"maas-api-revoke-sub-13a4195b-s8l6b\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.367700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.367676 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:01.746746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.746716 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:37:01.746871 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:01.746744 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5f246f8_3cb2_446d_a362_c90b466d16d3.slice/crio-b8f6304232a95d302050477c8595569e24018f8e480f6fcd2281ce8e41eaa28f WatchSource:0}: Error finding container b8f6304232a95d302050477c8595569e24018f8e480f6fcd2281ce8e41eaa28f: Status 404 returned error can't find the container with id b8f6304232a95d302050477c8595569e24018f8e480f6fcd2281ce8e41eaa28f Nov 01 14:37:01.746871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.746747 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" event={"ID":"22d76c85-4993-4586-b518-0be7e1a49232","Type":"ContainerDied","Data":"f09e3db564f561af564b1fc8e9c2c5d27a6d98bbe6dd9367dd499128cbb83d00"} Nov 01 14:37:01.746871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.746806 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f09e3db564f561af564b1fc8e9c2c5d27a6d98bbe6dd9367dd499128cbb83d00" Nov 01 14:37:01.751525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:01.751500 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b"] Nov 01 14:37:02.755902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:02.755867 5473 generic.go:356] "Generic (PLEG): container finished" podID="c5f246f8-3cb2-446d-a362-c90b466d16d3" containerID="c3b3be8bd3a4a2bc34132b108265372777377b43550518f0086464a15cc654c1" exitCode=0 Nov 01 14:37:02.756355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:02.755905 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" event={"ID":"c5f246f8-3cb2-446d-a362-c90b466d16d3","Type":"ContainerDied","Data":"c3b3be8bd3a4a2bc34132b108265372777377b43550518f0086464a15cc654c1"} Nov 01 14:37:02.756355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:02.755962 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" event={"ID":"c5f246f8-3cb2-446d-a362-c90b466d16d3","Type":"ContainerStarted","Data":"b8f6304232a95d302050477c8595569e24018f8e480f6fcd2281ce8e41eaa28f"} Nov 01 14:37:03.497746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.497716 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:03.913138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.913111 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:03.968130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.968097 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca\") pod \"c5f246f8-3cb2-446d-a362-c90b466d16d3\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " Nov 01 14:37:03.968309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.968151 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4\" (UniqueName: \"kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4\") pod \"c5f246f8-3cb2-446d-a362-c90b466d16d3\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " Nov 01 14:37:03.968309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.968268 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca\") pod \"c5f246f8-3cb2-446d-a362-c90b466d16d3\" (UID: \"c5f246f8-3cb2-446d-a362-c90b466d16d3\") " Nov 01 14:37:03.968600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.968574 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca" pod "c5f246f8-3cb2-446d-a362-c90b466d16d3" (UID: "c5f246f8-3cb2-446d-a362-c90b466d16d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:03.968705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.968657 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:03.970186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.970165 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca" pod "c5f246f8-3cb2-446d-a362-c90b466d16d3" (UID: "c5f246f8-3cb2-446d-a362-c90b466d16d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:03.970335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:03.970320 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4" pod "c5f246f8-3cb2-446d-a362-c90b466d16d3" (UID: "c5f246f8-3cb2-446d-a362-c90b466d16d3"). InnerVolumeSpecName "kube-api-access-2tjr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:04.069708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:04.069631 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5f246f8-3cb2-446d-a362-c90b466d16d3-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:04.069708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:04.069659 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tjr4\" (UniqueName: \"kubernetes.io/projected/c5f246f8-3cb2-446d-a362-c90b466d16d3-kube-api-access-2tjr4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:04.770476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:04.770436 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" event={"ID":"c5f246f8-3cb2-446d-a362-c90b466d16d3","Type":"ContainerDied","Data":"b8f6304232a95d302050477c8595569e24018f8e480f6fcd2281ce8e41eaa28f"} Nov 01 14:37:04.770476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:04.770471 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b" Nov 01 14:37:04.770690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:04.770519 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f6304232a95d302050477c8595569e24018f8e480f6fcd2281ce8e41eaa28f" Nov 01 14:37:05.676948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:05.676887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:05.893196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:05.893132 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:06.081517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:06.081484 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:07.691240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:07.688308 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:07.989306 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:07.988308 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:07.993241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:07.992933 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:08.129361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.129313 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:08.299241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.296911 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5"] Nov 01 14:37:08.337240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.322904 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c5f246f8-3cb2-446d-a362-c90b466d16d3" containerName="revoke-keys" Nov 01 14:37:08.337240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.322935 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f246f8-3cb2-446d-a362-c90b466d16d3" containerName="revoke-keys" Nov 01 14:37:08.337240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.323079 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f246f8-3cb2-446d-a362-c90b466d16d3" containerName="revoke-keys" Nov 01 14:37:08.337240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.327524 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.530157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.530118 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:37:08.790686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.684550 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:08.809422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.716504 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.761203 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9r5w\" (UniqueName: \"kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.812301 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.812368 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.790631 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.813556 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.845751 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:08.867273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.846284 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:08.908207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.907714 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:08.916016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.914836 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5"] Nov 01 14:37:08.918084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.917915 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9r5w\" (UniqueName: \"kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.918892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.918373 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.918892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.918542 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.923022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.922998 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:08.923398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:08.923341 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:09.001348 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.000201 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9r5w\" (UniqueName: \"kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w\") pod \"maas-api-revoke-sub-f40d8592-bpcx5\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:09.006988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.006661 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:09.017392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.017360 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:09.061441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.059742 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7"] Nov 01 14:37:09.065277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.065246 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7db58276-84dl7"] Nov 01 14:37:09.087552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.087515 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66666d98-3d16-4dad-8767-38cb9471b07e" path="/var/lib/kubelet/pods/66666d98-3d16-4dad-8767-38cb9471b07e/volumes" Nov 01 14:37:09.431181 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:09.431150 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb828607e_e25a_42dc_abd5_7376900bce3a.slice/crio-27d8064538578c2c41bddabed648251931fd06087cd5a53bd4b6b2554e34632d WatchSource:0}: Error finding container 27d8064538578c2c41bddabed648251931fd06087cd5a53bd4b6b2554e34632d: Status 404 returned error can't find the container with id 27d8064538578c2c41bddabed648251931fd06087cd5a53bd4b6b2554e34632d Nov 01 14:37:09.441086 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:09.441063 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5"] Nov 01 14:37:10.019245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:10.019197 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" event={"ID":"b828607e-e25a-42dc-abd5-7376900bce3a","Type":"ContainerStarted","Data":"8cf41cda46b60338735c5f7e1116569a79af0e643e9077f4cfdb5ffe0493466a"} Nov 01 14:37:10.019722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:10.019270 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" event={"ID":"b828607e-e25a-42dc-abd5-7376900bce3a","Type":"ContainerStarted","Data":"27d8064538578c2c41bddabed648251931fd06087cd5a53bd4b6b2554e34632d"} Nov 01 14:37:10.134195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:10.134145 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" podStartSLOduration=2.134131396 podStartE2EDuration="2.134131396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:10.132050228 +0000 UTC m=+1169.190202078" watchObservedRunningTime="2026-11-01 14:37:10.134131396 +0000 UTC m=+1169.192283245" Nov 01 14:37:10.723781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:10.723703 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:11.029226 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.029181 5473 generic.go:356] "Generic (PLEG): container finished" podID="b828607e-e25a-42dc-abd5-7376900bce3a" containerID="8cf41cda46b60338735c5f7e1116569a79af0e643e9077f4cfdb5ffe0493466a" exitCode=0 Nov 01 14:37:11.029623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.029248 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" event={"ID":"b828607e-e25a-42dc-abd5-7376900bce3a","Type":"ContainerDied","Data":"8cf41cda46b60338735c5f7e1116569a79af0e643e9077f4cfdb5ffe0493466a"} Nov 01 14:37:11.748016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.747983 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6"] Nov 01 14:37:11.752187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.752165 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.810677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.810639 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6"] Nov 01 14:37:11.855641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.855611 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.855797 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.855739 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9pz\" (UniqueName: \"kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.855873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.855823 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.956892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.956855 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.957050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.956957 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9pz\" (UniqueName: \"kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.957050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.957019 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.957666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.957646 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:11.959359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:11.959340 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:12.015151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.015117 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9pz\" (UniqueName: \"kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz\") pod \"maas-api-revoke-sub-9b5ed68a-xvfk6\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:12.066167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.066137 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:12.202426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.202402 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:12.260924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.260897 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w\" (UniqueName: \"kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w\") pod \"b828607e-e25a-42dc-abd5-7376900bce3a\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " Nov 01 14:37:12.261073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.260960 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca\") pod \"b828607e-e25a-42dc-abd5-7376900bce3a\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " Nov 01 14:37:12.261138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.261072 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca\") pod \"b828607e-e25a-42dc-abd5-7376900bce3a\" (UID: \"b828607e-e25a-42dc-abd5-7376900bce3a\") " Nov 01 14:37:12.261405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.261378 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca" pod "b828607e-e25a-42dc-abd5-7376900bce3a" (UID: "b828607e-e25a-42dc-abd5-7376900bce3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:12.261577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.261561 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:12.262863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.262842 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w" pod "b828607e-e25a-42dc-abd5-7376900bce3a" (UID: "b828607e-e25a-42dc-abd5-7376900bce3a"). InnerVolumeSpecName "kube-api-access-t9r5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:12.262919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.262892 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca" pod "b828607e-e25a-42dc-abd5-7376900bce3a" (UID: "b828607e-e25a-42dc-abd5-7376900bce3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:12.362969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.362897 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9r5w\" (UniqueName: \"kubernetes.io/projected/b828607e-e25a-42dc-abd5-7376900bce3a-kube-api-access-t9r5w\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:12.362969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.362925 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b828607e-e25a-42dc-abd5-7376900bce3a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:12.437805 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:12.437774 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd8ce8f_6bcb_474b_91ea_4652f1959a30.slice/crio-2fc420128decf7961ec6ad4949934d4544205d58fdcfbf1eb3dc9719a6588a6e WatchSource:0}: Error finding container 2fc420128decf7961ec6ad4949934d4544205d58fdcfbf1eb3dc9719a6588a6e: Status 404 returned error can't find the container with id 2fc420128decf7961ec6ad4949934d4544205d58fdcfbf1eb3dc9719a6588a6e Nov 01 14:37:12.442163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:12.442139 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6"] Nov 01 14:37:13.045049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.045016 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" event={"ID":"2dd8ce8f-6bcb-474b-91ea-4652f1959a30","Type":"ContainerStarted","Data":"0c813b8ec0821e98653076b25e1437c2feb7ec48a812fef4047fb096d624a58c"} Nov 01 14:37:13.045233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.045082 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" event={"ID":"2dd8ce8f-6bcb-474b-91ea-4652f1959a30","Type":"ContainerStarted","Data":"2fc420128decf7961ec6ad4949934d4544205d58fdcfbf1eb3dc9719a6588a6e"} Nov 01 14:37:13.046634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.046610 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" Nov 01 14:37:13.046757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.046654 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5" event={"ID":"b828607e-e25a-42dc-abd5-7376900bce3a","Type":"ContainerDied","Data":"27d8064538578c2c41bddabed648251931fd06087cd5a53bd4b6b2554e34632d"} Nov 01 14:37:13.046757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.046700 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d8064538578c2c41bddabed648251931fd06087cd5a53bd4b6b2554e34632d" Nov 01 14:37:13.161687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.161638 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" podStartSLOduration=2.161623806 podStartE2EDuration="2.161623806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:13.158708616 +0000 UTC m=+1172.216860466" watchObservedRunningTime="2026-11-01 14:37:13.161623806 +0000 UTC m=+1172.219775653" Nov 01 14:37:13.389436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:13.389400 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:14.055777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:14.055747 5473 generic.go:356] "Generic (PLEG): container finished" podID="2dd8ce8f-6bcb-474b-91ea-4652f1959a30" containerID="0c813b8ec0821e98653076b25e1437c2feb7ec48a812fef4047fb096d624a58c" exitCode=0 Nov 01 14:37:14.055939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:14.055834 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" event={"ID":"2dd8ce8f-6bcb-474b-91ea-4652f1959a30","Type":"ContainerDied","Data":"0c813b8ec0821e98653076b25e1437c2feb7ec48a812fef4047fb096d624a58c"} Nov 01 14:37:15.204302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.204279 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:15.298798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.298769 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca\") pod \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " Nov 01 14:37:15.298912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.298820 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca\") pod \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " Nov 01 14:37:15.298912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.298855 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz\" (UniqueName: \"kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz\") pod \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\" (UID: \"2dd8ce8f-6bcb-474b-91ea-4652f1959a30\") " Nov 01 14:37:15.299156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.299133 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca" pod "2dd8ce8f-6bcb-474b-91ea-4652f1959a30" (UID: "2dd8ce8f-6bcb-474b-91ea-4652f1959a30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:15.299350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.299331 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:15.300887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.300864 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca" pod "2dd8ce8f-6bcb-474b-91ea-4652f1959a30" (UID: "2dd8ce8f-6bcb-474b-91ea-4652f1959a30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:15.300965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.300891 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz" pod "2dd8ce8f-6bcb-474b-91ea-4652f1959a30" (UID: "2dd8ce8f-6bcb-474b-91ea-4652f1959a30"). InnerVolumeSpecName "kube-api-access-5d9pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:15.400567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.400498 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:15.400567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:15.400525 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9pz\" (UniqueName: \"kubernetes.io/projected/2dd8ce8f-6bcb-474b-91ea-4652f1959a30-kube-api-access-5d9pz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:16.071812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:16.071783 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" Nov 01 14:37:16.072003 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:16.071826 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6" event={"ID":"2dd8ce8f-6bcb-474b-91ea-4652f1959a30","Type":"ContainerDied","Data":"2fc420128decf7961ec6ad4949934d4544205d58fdcfbf1eb3dc9719a6588a6e"} Nov 01 14:37:16.072003 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:16.071873 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fc420128decf7961ec6ad4949934d4544205d58fdcfbf1eb3dc9719a6588a6e" Nov 01 14:37:17.786174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:17.786131 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:17.942458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:17.942424 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:21.991392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:21.991293 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:22.015330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:22.015286 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:22.132442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:22.131290 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:22.541367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:22.541320 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 14:37:23.020299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:22.994644 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:23.020299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:22.995394 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:37:23.020299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.017309 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:23.020299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.017695 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:23.020299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.018858 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:23.262403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.262358 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 01 14:37:23.262999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.262968 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": context deadline exceeded" Nov 01 14:37:23.296308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.265795 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:37:23.566631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.566534 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:23.600171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.345335 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:23.600858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.600817 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:23.681239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.659500 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Nov 01 14:37:23.681239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.660469 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:37:23.915836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.915607 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:23.916280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.916244 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:23.970475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:23.970442 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:37:24.001780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:24.001752 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:24.002034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:24.002023 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:37:24.014844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:24.014791 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:24.014975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:24.014889 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:24.275665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:24.275637 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:37:24.498679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:24.498641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:26.425386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.425357 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx"] Nov 01 14:37:26.425888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.425875 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b828607e-e25a-42dc-abd5-7376900bce3a" containerName="revoke-keys" Nov 01 14:37:26.425936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.425889 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b828607e-e25a-42dc-abd5-7376900bce3a" containerName="revoke-keys" Nov 01 14:37:26.425936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.425926 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2dd8ce8f-6bcb-474b-91ea-4652f1959a30" containerName="revoke-keys" Nov 01 14:37:26.425936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.425931 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd8ce8f-6bcb-474b-91ea-4652f1959a30" containerName="revoke-keys" Nov 01 14:37:26.426024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.426018 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b828607e-e25a-42dc-abd5-7376900bce3a" containerName="revoke-keys" Nov 01 14:37:26.426057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.426032 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd8ce8f-6bcb-474b-91ea-4652f1959a30" containerName="revoke-keys" Nov 01 14:37:26.431150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.431133 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.482376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.482349 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx"] Nov 01 14:37:26.523158 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.523131 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.523309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.523162 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.523400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.523382 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9br2b\" (UniqueName: \"kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.527038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.527020 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:37:26.527113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.527098 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:37:26.624465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.624437 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9br2b\" (UniqueName: \"kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.624613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.624523 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.624613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.624543 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.625146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.625122 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.626887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.626869 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.680695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.680641 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9br2b\" (UniqueName: \"kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b\") pod \"maas-api-revoke-sub-fb9db681-985rx\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:26.744190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:26.744165 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:27.114100 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:27.114072 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8e4c42_51e1_4b09_8988_928f889b17b8.slice/crio-4095cd45f71a36387e0592b2dd0f3bb6b24d2f02e143ce99a0df6db1b7c10382 WatchSource:0}: Error finding container 4095cd45f71a36387e0592b2dd0f3bb6b24d2f02e143ce99a0df6db1b7c10382: Status 404 returned error can't find the container with id 4095cd45f71a36387e0592b2dd0f3bb6b24d2f02e143ce99a0df6db1b7c10382 Nov 01 14:37:27.117545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:27.117523 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx"] Nov 01 14:37:27.909768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:27.909733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" event={"ID":"1d8e4c42-51e1-4b09-8988-928f889b17b8","Type":"ContainerStarted","Data":"c3331878798f952d4dc51b8b18acedead13ac3e631d70f52ad036d9fb00337b6"} Nov 01 14:37:27.910137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:27.909815 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" event={"ID":"1d8e4c42-51e1-4b09-8988-928f889b17b8","Type":"ContainerStarted","Data":"4095cd45f71a36387e0592b2dd0f3bb6b24d2f02e143ce99a0df6db1b7c10382"} Nov 01 14:37:28.025867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:28.025821 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" podStartSLOduration=2.025808305 podStartE2EDuration="2.025808305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:28.022599984 +0000 UTC m=+1187.080751853" watchObservedRunningTime="2026-11-01 14:37:28.025808305 +0000 UTC m=+1187.083960155" Nov 01 14:37:28.897990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:28.897961 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:28.923927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:28.923896 5473 generic.go:356] "Generic (PLEG): container finished" podID="1d8e4c42-51e1-4b09-8988-928f889b17b8" containerID="c3331878798f952d4dc51b8b18acedead13ac3e631d70f52ad036d9fb00337b6" exitCode=0 Nov 01 14:37:28.924415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:28.923981 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" event={"ID":"1d8e4c42-51e1-4b09-8988-928f889b17b8","Type":"ContainerDied","Data":"c3331878798f952d4dc51b8b18acedead13ac3e631d70f52ad036d9fb00337b6"} Nov 01 14:37:30.077322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.077299 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:30.170873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.170843 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca\") pod \"1d8e4c42-51e1-4b09-8988-928f889b17b8\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " Nov 01 14:37:30.171018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.170890 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b\" (UniqueName: \"kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b\") pod \"1d8e4c42-51e1-4b09-8988-928f889b17b8\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " Nov 01 14:37:30.171058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.171044 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca\") pod \"1d8e4c42-51e1-4b09-8988-928f889b17b8\" (UID: \"1d8e4c42-51e1-4b09-8988-928f889b17b8\") " Nov 01 14:37:30.171207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.171185 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca" pod "1d8e4c42-51e1-4b09-8988-928f889b17b8" (UID: "1d8e4c42-51e1-4b09-8988-928f889b17b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:30.171560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.171539 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:30.173036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.173006 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b" pod "1d8e4c42-51e1-4b09-8988-928f889b17b8" (UID: "1d8e4c42-51e1-4b09-8988-928f889b17b8"). InnerVolumeSpecName "kube-api-access-9br2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:30.173134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.173059 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca" pod "1d8e4c42-51e1-4b09-8988-928f889b17b8" (UID: "1d8e4c42-51e1-4b09-8988-928f889b17b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:30.272257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.272230 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d8e4c42-51e1-4b09-8988-928f889b17b8-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:30.272257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.272256 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9br2b\" (UniqueName: \"kubernetes.io/projected/1d8e4c42-51e1-4b09-8988-928f889b17b8-kube-api-access-9br2b\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:30.523566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.523537 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn"] Nov 01 14:37:30.524131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.524115 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1d8e4c42-51e1-4b09-8988-928f889b17b8" containerName="revoke-keys" Nov 01 14:37:30.524131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.524131 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8e4c42-51e1-4b09-8988-928f889b17b8" containerName="revoke-keys" Nov 01 14:37:30.524305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.524241 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8e4c42-51e1-4b09-8988-928f889b17b8" containerName="revoke-keys" Nov 01 14:37:30.527903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.527883 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.574501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.574475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.574643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.574535 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhch\" (UniqueName: \"kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.574643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.574621 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.612203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.612171 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn"] Nov 01 14:37:30.676393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.676368 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.676538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.676432 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhch\" (UniqueName: \"kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.676538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.676501 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.677057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.677033 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.678834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.678812 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.732973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.732941 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhch\" (UniqueName: \"kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch\") pod \"maas-api-revoke-sub-ab0c7b31-dspgn\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.843211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.843146 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:30.945277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.940952 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:37:30.945277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.940976 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" event={"ID":"1d8e4c42-51e1-4b09-8988-928f889b17b8","Type":"ContainerDied","Data":"4095cd45f71a36387e0592b2dd0f3bb6b24d2f02e143ce99a0df6db1b7c10382"} Nov 01 14:37:30.945277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:30.941026 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4095cd45f71a36387e0592b2dd0f3bb6b24d2f02e143ce99a0df6db1b7c10382" Nov 01 14:37:31.107752 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:37:31.107681 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:37:31.210514 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:31.210485 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a250fb2_899b_4a51_a3f9_5d561451e541.slice/crio-682f3324ce8171c40b124af6c34432ca86a836f8d83b7009b55e84c690544621 WatchSource:0}: Error finding container 682f3324ce8171c40b124af6c34432ca86a836f8d83b7009b55e84c690544621: Status 404 returned error can't find the container with id 682f3324ce8171c40b124af6c34432ca86a836f8d83b7009b55e84c690544621 Nov 01 14:37:31.215203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:31.215184 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn"] Nov 01 14:37:31.950176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:31.950143 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" event={"ID":"7a250fb2-899b-4a51-a3f9-5d561451e541","Type":"ContainerStarted","Data":"b1140351160bac2f06cfacb95882759b151c4acf347a7da0db4bdeb2fcedba4b"} Nov 01 14:37:31.950355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:31.950204 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" event={"ID":"7a250fb2-899b-4a51-a3f9-5d561451e541","Type":"ContainerStarted","Data":"682f3324ce8171c40b124af6c34432ca86a836f8d83b7009b55e84c690544621"} Nov 01 14:37:32.056794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:32.056765 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g"] Nov 01 14:37:32.060479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:32.060456 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5a5f8be-56c6g"] Nov 01 14:37:32.098152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:32.098110 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" podStartSLOduration=2.09809876 podStartE2EDuration="2.09809876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:32.096031525 +0000 UTC m=+1191.154183372" watchObservedRunningTime="2026-11-01 14:37:32.09809876 +0000 UTC m=+1191.156250655" Nov 01 14:37:32.959477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:32.959442 5473 generic.go:356] "Generic (PLEG): container finished" podID="7a250fb2-899b-4a51-a3f9-5d561451e541" containerID="b1140351160bac2f06cfacb95882759b151c4acf347a7da0db4bdeb2fcedba4b" exitCode=0 Nov 01 14:37:32.959854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:32.959490 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" event={"ID":"7a250fb2-899b-4a51-a3f9-5d561451e541","Type":"ContainerDied","Data":"b1140351160bac2f06cfacb95882759b151c4acf347a7da0db4bdeb2fcedba4b"} Nov 01 14:37:33.081939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:33.081908 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb0c20b-3ca8-4ce0-9370-65035800d1d2" path="/var/lib/kubelet/pods/ecb0c20b-3ca8-4ce0-9370-65035800d1d2/volumes" Nov 01 14:37:33.342493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:33.342431 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:34.052249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.052055 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn"] Nov 01 14:37:34.098238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.098193 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1255447-mghzn"] Nov 01 14:37:34.116227 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.116186 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:34.214663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.214636 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca\") pod \"7a250fb2-899b-4a51-a3f9-5d561451e541\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " Nov 01 14:37:34.214799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.214682 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch\" (UniqueName: \"kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch\") pod \"7a250fb2-899b-4a51-a3f9-5d561451e541\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " Nov 01 14:37:34.214799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.214778 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca\") pod \"7a250fb2-899b-4a51-a3f9-5d561451e541\" (UID: \"7a250fb2-899b-4a51-a3f9-5d561451e541\") " Nov 01 14:37:34.215022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.214996 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca" pod "7a250fb2-899b-4a51-a3f9-5d561451e541" (UID: "7a250fb2-899b-4a51-a3f9-5d561451e541"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:34.215172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.215150 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:34.216792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.216764 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca" pod "7a250fb2-899b-4a51-a3f9-5d561451e541" (UID: "7a250fb2-899b-4a51-a3f9-5d561451e541"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:34.216911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.216796 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch" pod "7a250fb2-899b-4a51-a3f9-5d561451e541" (UID: "7a250fb2-899b-4a51-a3f9-5d561451e541"). InnerVolumeSpecName "kube-api-access-gkhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:34.316559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.316531 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkhch\" (UniqueName: \"kubernetes.io/projected/7a250fb2-899b-4a51-a3f9-5d561451e541-kube-api-access-gkhch\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:34.316559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.316559 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a250fb2-899b-4a51-a3f9-5d561451e541-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:34.982735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.976971 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" Nov 01 14:37:34.982735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.977074 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn" event={"ID":"7a250fb2-899b-4a51-a3f9-5d561451e541","Type":"ContainerDied","Data":"682f3324ce8171c40b124af6c34432ca86a836f8d83b7009b55e84c690544621"} Nov 01 14:37:34.982735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:34.977113 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682f3324ce8171c40b124af6c34432ca86a836f8d83b7009b55e84c690544621" Nov 01 14:37:35.081581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:35.081552 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f" path="/var/lib/kubelet/pods/8c60fbf3-3f00-46bc-bd7c-d68f3aa36d7f/volumes" Nov 01 14:37:37.102745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.102704 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg"] Nov 01 14:37:37.103264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.103250 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7a250fb2-899b-4a51-a3f9-5d561451e541" containerName="revoke-keys" Nov 01 14:37:37.103317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.103265 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a250fb2-899b-4a51-a3f9-5d561451e541" containerName="revoke-keys" Nov 01 14:37:37.103389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.103379 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a250fb2-899b-4a51-a3f9-5d561451e541" containerName="revoke-keys" Nov 01 14:37:37.108004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.107981 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.159950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.159914 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg"] Nov 01 14:37:37.204196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.204164 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:37:37.204365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.204209 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:37:37.251532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.251504 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.251684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.251635 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.251768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.251751 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7cn\" (UniqueName: \"kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.352918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.352853 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.353040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.352934 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.353040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.352986 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7cn\" (UniqueName: \"kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.353505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.353484 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.355248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.355201 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.409921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.409899 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7cn\" (UniqueName: \"kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn\") pod \"maas-api-revoke-sub-be50e8e3-x5fcg\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.421749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.421728 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:37.708202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.708121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:37.791740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:37.791706 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684c9c25_86c7_4307_8b1d_61b44a49c899.slice/crio-a80fb0f323ce7a39e17c769fae861494bfa220a8c0db26b1173cba19f94cfc59 WatchSource:0}: Error finding container a80fb0f323ce7a39e17c769fae861494bfa220a8c0db26b1173cba19f94cfc59: Status 404 returned error can't find the container with id a80fb0f323ce7a39e17c769fae861494bfa220a8c0db26b1173cba19f94cfc59 Nov 01 14:37:37.796352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:37.796147 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg"] Nov 01 14:37:38.001423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:38.001335 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" event={"ID":"684c9c25-86c7-4307-8b1d-61b44a49c899","Type":"ContainerStarted","Data":"a80fb0f323ce7a39e17c769fae861494bfa220a8c0db26b1173cba19f94cfc59"} Nov 01 14:37:39.010325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:39.010243 5473 generic.go:356] "Generic (PLEG): container finished" podID="684c9c25-86c7-4307-8b1d-61b44a49c899" containerID="ccf0454909e74c8a75dd80574a4963c84eaa80b60bd52081965e2969b86293c0" exitCode=0 Nov 01 14:37:39.010760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:39.010328 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" event={"ID":"684c9c25-86c7-4307-8b1d-61b44a49c899","Type":"ContainerDied","Data":"ccf0454909e74c8a75dd80574a4963c84eaa80b60bd52081965e2969b86293c0"} Nov 01 14:37:40.588240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.570863 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:40.626770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.626684 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca\") pod \"684c9c25-86c7-4307-8b1d-61b44a49c899\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " Nov 01 14:37:40.627399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.627375 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn\" (UniqueName: \"kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn\") pod \"684c9c25-86c7-4307-8b1d-61b44a49c899\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " Nov 01 14:37:40.628060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.628041 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca\") pod \"684c9c25-86c7-4307-8b1d-61b44a49c899\" (UID: \"684c9c25-86c7-4307-8b1d-61b44a49c899\") " Nov 01 14:37:40.643305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.633396 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca" pod "684c9c25-86c7-4307-8b1d-61b44a49c899" (UID: "684c9c25-86c7-4307-8b1d-61b44a49c899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:40.664430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.664398 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca" pod "684c9c25-86c7-4307-8b1d-61b44a49c899" (UID: "684c9c25-86c7-4307-8b1d-61b44a49c899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:40.666708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.666683 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn" pod "684c9c25-86c7-4307-8b1d-61b44a49c899" (UID: "684c9c25-86c7-4307-8b1d-61b44a49c899"). InnerVolumeSpecName "kube-api-access-np7cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:40.762936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.762898 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:40.763176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.763156 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/684c9c25-86c7-4307-8b1d-61b44a49c899-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:40.815255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:40.809279 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np7cn\" (UniqueName: \"kubernetes.io/projected/684c9c25-86c7-4307-8b1d-61b44a49c899-kube-api-access-np7cn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:41.042890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:41.042853 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" event={"ID":"684c9c25-86c7-4307-8b1d-61b44a49c899","Type":"ContainerDied","Data":"a80fb0f323ce7a39e17c769fae861494bfa220a8c0db26b1173cba19f94cfc59"} Nov 01 14:37:41.043189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:41.043169 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a80fb0f323ce7a39e17c769fae861494bfa220a8c0db26b1173cba19f94cfc59" Nov 01 14:37:41.043388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:41.043373 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:37:41.990245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:41.989645 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 01 14:37:41.994237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:41.990916 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 01 14:37:41.994237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:41.991694 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.090606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.087106 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:37:42.304237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.289705 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.462248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.442802 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:42.462248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.442890 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.586315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.576389 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt"] Nov 01 14:37:42.586315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.577310 5473 cpu_manager.go:414] "containerMap: removing container" podUID="684c9c25-86c7-4307-8b1d-61b44a49c899" containerName="revoke-keys" Nov 01 14:37:42.586315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.577332 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="684c9c25-86c7-4307-8b1d-61b44a49c899" containerName="revoke-keys" Nov 01 14:37:42.586315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.577556 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="684c9c25-86c7-4307-8b1d-61b44a49c899" containerName="revoke-keys" Nov 01 14:37:42.677238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.661818 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt"] Nov 01 14:37:42.677238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.661966 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.731239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.728964 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:37:42.731239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.729741 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:37:42.735242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.733534 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.735242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.733584 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5tl\" (UniqueName: \"kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.735242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.733654 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.739240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.738380 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:37:42.847286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.847056 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.847286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.847120 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5tl\" (UniqueName: \"kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.847286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.847203 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.858283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.848600 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.866249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.859723 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:42.992478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.990135 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.992478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.990814 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:37:42.992478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.991804 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.992478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.992158 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.993705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.993678 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:42.994130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.994112 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:37:42.994318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.994305 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:42.996893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:42.996872 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:43.299370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.272909 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:43.438229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.432297 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:43.454291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.447716 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5tl\" (UniqueName: \"kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl\") pod \"maas-api-revoke-sub-6aa0eae4-2smrt\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:43.455007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.454772 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:43.594613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.594564 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:37:43.594613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.594620 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:37:43.594829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.594648 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s"] Nov 01 14:37:43.601374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.601251 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:37:43.601374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.601285 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:43.601374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.601302 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s"] Nov 01 14:37:43.601374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.601328 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:37:43.602041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.601906 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.696942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.696387 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j877\" (UniqueName: \"kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.696942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.696694 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.696942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.696860 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.798348 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.798315 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.798525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.798488 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.798621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.798605 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j877\" (UniqueName: \"kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.799208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.799178 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.800985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.800938 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.855752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.855689 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j877\" (UniqueName: \"kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877\") pod \"maas-api-revoke-sub-905256ee-szc6s\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.929209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.929178 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:43.971250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:43.970528 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt"] Nov 01 14:37:44.325613 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:44.325575 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271c7068_ade3_4b93_a0be_6eaac7715343.slice/crio-62977e977933ad24a54833a3a7449a4c697be8c0e682d9ec71d2632cc8ec82a4 WatchSource:0}: Error finding container 62977e977933ad24a54833a3a7449a4c697be8c0e682d9ec71d2632cc8ec82a4: Status 404 returned error can't find the container with id 62977e977933ad24a54833a3a7449a4c697be8c0e682d9ec71d2632cc8ec82a4 Nov 01 14:37:44.327842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:44.327811 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s"] Nov 01 14:37:44.603827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:44.603790 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" event={"ID":"6a527c83-bab9-4e0c-894c-4fe97b8591c1","Type":"ContainerStarted","Data":"6908533dd9d56cb0017492b584e3dbc4b210d66fbed55b1929454577c825c4ba"} Nov 01 14:37:44.603998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:44.603852 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" event={"ID":"6a527c83-bab9-4e0c-894c-4fe97b8591c1","Type":"ContainerStarted","Data":"bf7920f1f5da089855096d795cef509bb3ff7417fb5919909efd0b449b783e77"} Nov 01 14:37:44.605134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:44.605110 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" event={"ID":"271c7068-ade3-4b93-a0be-6eaac7715343","Type":"ContainerStarted","Data":"62977e977933ad24a54833a3a7449a4c697be8c0e682d9ec71d2632cc8ec82a4"} Nov 01 14:37:44.720291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:44.720227 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" podStartSLOduration=3.720193546 podStartE2EDuration="3.720193546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:44.718042903 +0000 UTC m=+1203.776194765" watchObservedRunningTime="2026-11-01 14:37:44.720193546 +0000 UTC m=+1203.778345397" Nov 01 14:37:45.642826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:45.642788 5473 generic.go:356] "Generic (PLEG): container finished" podID="6a527c83-bab9-4e0c-894c-4fe97b8591c1" containerID="6908533dd9d56cb0017492b584e3dbc4b210d66fbed55b1929454577c825c4ba" exitCode=0 Nov 01 14:37:45.643311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:45.642872 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" event={"ID":"6a527c83-bab9-4e0c-894c-4fe97b8591c1","Type":"ContainerDied","Data":"6908533dd9d56cb0017492b584e3dbc4b210d66fbed55b1929454577c825c4ba"} Nov 01 14:37:45.644862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:45.644838 5473 generic.go:356] "Generic (PLEG): container finished" podID="271c7068-ade3-4b93-a0be-6eaac7715343" containerID="f363bdd105e49559dc59ead089f271f1d9b1e61fb41e4b6877bdfbc06df44d84" exitCode=0 Nov 01 14:37:45.644973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:45.644865 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" event={"ID":"271c7068-ade3-4b93-a0be-6eaac7715343","Type":"ContainerDied","Data":"f363bdd105e49559dc59ead089f271f1d9b1e61fb41e4b6877bdfbc06df44d84"} Nov 01 14:37:45.987041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:45.986962 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv"] Nov 01 14:37:45.991531 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:45.991508 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.003897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.003871 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz"] Nov 01 14:37:46.008629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.008596 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062469 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062519 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062549 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062600 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062642 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062665 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062722 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062744 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062771 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062817 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062842 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062880 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpb6j\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.062919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062905 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjwf\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.063725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.062950 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.070400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.063981 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.070400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.064059 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.070400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.064116 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.070400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.064149 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.076271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.073483 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv"] Nov 01 14:37:46.165249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.165197 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.166665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.166639 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.167191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.167171 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.167764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.167743 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.168686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.168665 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.169092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.169072 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.169705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.169686 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.170579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.170559 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.171495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.171475 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpb6j\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.171893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.171873 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjwf\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.172338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.172315 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.172812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.172794 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.173186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173159 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.173373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173261 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.173373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173298 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.173488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173409 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.173488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173439 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.173488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173472 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.174241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.170304 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.174241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.168594 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.174241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.171402 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.174241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173954 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.174241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.173089 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.174241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.167639 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.174598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.172699 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.174598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.174556 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.174704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.174591 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.174767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.174735 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.176179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.176157 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.176415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.176390 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.177359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.177329 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.177957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.177929 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.178286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.178266 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz"] Nov 01 14:37:46.225821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.225792 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.267788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.267752 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.267788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.267774 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjwf\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.267977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.267873 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpb6j\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j\") pod \"e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.306947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.306918 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:46.327451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.326452 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:46.773488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.773451 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv"] Nov 01 14:37:46.773931 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:46.773797 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fde8df_a3c5_470e_9e6d_0df9fa6d6576.slice/crio-60d71a75fa667082faa951d8047c58d72b55342c887184b9bb10c2f8420023d9 WatchSource:0}: Error finding container 60d71a75fa667082faa951d8047c58d72b55342c887184b9bb10c2f8420023d9: Status 404 returned error can't find the container with id 60d71a75fa667082faa951d8047c58d72b55342c887184b9bb10c2f8420023d9 Nov 01 14:37:46.776334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.776293 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:37:46.776445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.776384 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:37:46.776445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.776426 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:37:46.821284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.821259 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz"] Nov 01 14:37:46.826436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.824948 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:37:46.826436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.825035 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:37:46.826436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.825082 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:37:46.906662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.906635 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:46.927050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.926191 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:47.002246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.999778 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl\" (UniqueName: \"kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl\") pod \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " Nov 01 14:37:47.002246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.999826 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877\" (UniqueName: \"kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877\") pod \"271c7068-ade3-4b93-a0be-6eaac7715343\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " Nov 01 14:37:47.002246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.999904 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca\") pod \"271c7068-ade3-4b93-a0be-6eaac7715343\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " Nov 01 14:37:47.002246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:46.999979 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca\") pod \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " Nov 01 14:37:47.002246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.000078 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca\") pod \"271c7068-ade3-4b93-a0be-6eaac7715343\" (UID: \"271c7068-ade3-4b93-a0be-6eaac7715343\") " Nov 01 14:37:47.002246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.000114 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca\") pod \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\" (UID: \"6a527c83-bab9-4e0c-894c-4fe97b8591c1\") " Nov 01 14:37:47.006686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.006651 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca" pod "6a527c83-bab9-4e0c-894c-4fe97b8591c1" (UID: "6a527c83-bab9-4e0c-894c-4fe97b8591c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:47.008852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.007679 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca" pod "271c7068-ade3-4b93-a0be-6eaac7715343" (UID: "271c7068-ade3-4b93-a0be-6eaac7715343"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:37:47.019938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.019910 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca" pod "6a527c83-bab9-4e0c-894c-4fe97b8591c1" (UID: "6a527c83-bab9-4e0c-894c-4fe97b8591c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:47.020111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.020088 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877" pod "271c7068-ade3-4b93-a0be-6eaac7715343" (UID: "271c7068-ade3-4b93-a0be-6eaac7715343"). InnerVolumeSpecName "kube-api-access-7j877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:47.029281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.027159 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca" pod "271c7068-ade3-4b93-a0be-6eaac7715343" (UID: "271c7068-ade3-4b93-a0be-6eaac7715343"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:37:47.029281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.027316 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl" pod "6a527c83-bab9-4e0c-894c-4fe97b8591c1" (UID: "6a527c83-bab9-4e0c-894c-4fe97b8591c1"). InnerVolumeSpecName "kube-api-access-dt5tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:37:47.056299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.054399 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w"] Nov 01 14:37:47.060945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.060883 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0a8cc0e-4vd2w"] Nov 01 14:37:47.087989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.087953 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f27e14-07db-4419-8ae0-de0611cfa937" path="/var/lib/kubelet/pods/b2f27e14-07db-4419-8ae0-de0611cfa937/volumes" Nov 01 14:37:47.107469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.104250 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:47.107469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.104280 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:47.107469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.104298 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt5tl\" (UniqueName: \"kubernetes.io/projected/6a527c83-bab9-4e0c-894c-4fe97b8591c1-kube-api-access-dt5tl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:47.107469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.104314 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j877\" (UniqueName: \"kubernetes.io/projected/271c7068-ade3-4b93-a0be-6eaac7715343-kube-api-access-7j877\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:47.107469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.104330 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271c7068-ade3-4b93-a0be-6eaac7715343-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:47.107469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.104346 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a527c83-bab9-4e0c-894c-4fe97b8591c1-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:37:47.507637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.507518 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:47.699510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.699466 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" event={"ID":"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6","Type":"ContainerStarted","Data":"00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5"} Nov 01 14:37:47.699715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.699537 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" event={"ID":"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6","Type":"ContainerStarted","Data":"8665d1c2c84a25c66798c1e32c160c67637f985f44608daa3070765afc098fb0"} Nov 01 14:37:47.701130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.701104 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" Nov 01 14:37:47.701280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.701142 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt" event={"ID":"6a527c83-bab9-4e0c-894c-4fe97b8591c1","Type":"ContainerDied","Data":"bf7920f1f5da089855096d795cef509bb3ff7417fb5919909efd0b449b783e77"} Nov 01 14:37:47.701280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.701187 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7920f1f5da089855096d795cef509bb3ff7417fb5919909efd0b449b783e77" Nov 01 14:37:47.702872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.702846 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" event={"ID":"271c7068-ade3-4b93-a0be-6eaac7715343","Type":"ContainerDied","Data":"62977e977933ad24a54833a3a7449a4c697be8c0e682d9ec71d2632cc8ec82a4"} Nov 01 14:37:47.702991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.702893 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62977e977933ad24a54833a3a7449a4c697be8c0e682d9ec71d2632cc8ec82a4" Nov 01 14:37:47.702991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.702855 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s" Nov 01 14:37:47.707185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.707155 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" event={"ID":"17fde8df-a3c5-470e-9e6d-0df9fa6d6576","Type":"ContainerStarted","Data":"88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979"} Nov 01 14:37:47.707285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.707198 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" event={"ID":"17fde8df-a3c5-470e-9e6d-0df9fa6d6576","Type":"ContainerStarted","Data":"60d71a75fa667082faa951d8047c58d72b55342c887184b9bb10c2f8420023d9"} Nov 01 14:37:47.825706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.824363 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podStartSLOduration=2.824341702 podStartE2EDuration="2.824341702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:47.81776458 +0000 UTC m=+1206.875916429" watchObservedRunningTime="2026-11-01 14:37:47.824341702 +0000 UTC m=+1206.882493552" Nov 01 14:37:47.991254 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:47.991176 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podStartSLOduration=2.991158845 podStartE2EDuration="2.991158845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:47.988599674 +0000 UTC m=+1207.046751536" watchObservedRunningTime="2026-11-01 14:37:47.991158845 +0000 UTC m=+1207.049310696" Nov 01 14:37:48.307882 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:48.307848 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:48.309547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:48.309507 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:48.309675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:48.309568 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:48.334296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:48.332664 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:48.350243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:48.336343 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:48.350243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:48.336436 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:49.308459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.308170 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:49.308919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.308887 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:49.338363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.338329 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:49.339018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.338985 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:49.356594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.356563 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc"] Nov 01 14:37:49.357700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.357670 5473 cpu_manager.go:414] "containerMap: removing container" podUID="271c7068-ade3-4b93-a0be-6eaac7715343" containerName="revoke-keys" Nov 01 14:37:49.357853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.357837 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271c7068-ade3-4b93-a0be-6eaac7715343" containerName="revoke-keys" Nov 01 14:37:49.358001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.357988 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6a527c83-bab9-4e0c-894c-4fe97b8591c1" containerName="revoke-keys" Nov 01 14:37:49.358082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.358071 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a527c83-bab9-4e0c-894c-4fe97b8591c1" containerName="revoke-keys" Nov 01 14:37:49.358374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.358356 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a527c83-bab9-4e0c-894c-4fe97b8591c1" containerName="revoke-keys" Nov 01 14:37:49.358488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.358477 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="271c7068-ade3-4b93-a0be-6eaac7715343" containerName="revoke-keys" Nov 01 14:37:49.368787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.368764 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.383589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.371208 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:49.391260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.387907 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc"] Nov 01 14:37:49.424249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.422532 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1f8e2c\"/\"distinct-2-w0-1f8e2c-kserve-self-signed-certs\"" Nov 01 14:37:49.492270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.492027 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.492270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.492254 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.492536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.492308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhv9\" (UniqueName: \"kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.492536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.492371 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.492536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.492439 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.492536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.492467 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.566293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.546188 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:49.604023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.603967 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.604441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.604418 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.604613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.604597 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhv9\" (UniqueName: \"kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.604760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.604745 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.604892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.604879 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.604989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.604975 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.606261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.606181 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.608279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.606918 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.608381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.607135 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.611079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.611058 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.615438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.615413 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.687534 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.687480 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhv9\" (UniqueName: \"kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9\") pod \"distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:49.708659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:49.708387 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:50.317044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.317010 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:50.317392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.317072 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:50.318755 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:50.318726 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89542591_7b5b_47d1_aa56_fed62b5a2290.slice/crio-899baaba9d8ccd5d714107fc8aa193b20d6a89c4c88e4aabebc56f3dd12c1640 WatchSource:0}: Error finding container 899baaba9d8ccd5d714107fc8aa193b20d6a89c4c88e4aabebc56f3dd12c1640: Status 404 returned error can't find the container with id 899baaba9d8ccd5d714107fc8aa193b20d6a89c4c88e4aabebc56f3dd12c1640 Nov 01 14:37:50.328580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.328553 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc"] Nov 01 14:37:50.340244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.337539 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:50.340244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.337601 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:50.903541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.903508 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" event={"ID":"89542591-7b5b-47d1-aa56-fed62b5a2290","Type":"ContainerStarted","Data":"d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5"} Nov 01 14:37:50.903541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:50.903547 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" event={"ID":"89542591-7b5b-47d1-aa56-fed62b5a2290","Type":"ContainerStarted","Data":"899baaba9d8ccd5d714107fc8aa193b20d6a89c4c88e4aabebc56f3dd12c1640"} Nov 01 14:37:51.020100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:51.020041 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podStartSLOduration=2.020027208 podStartE2EDuration="2.020027208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:51.017818886 +0000 UTC m=+1210.075970737" watchObservedRunningTime="2026-11-01 14:37:51.020027208 +0000 UTC m=+1210.078179057" Nov 01 14:37:51.308050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:51.307980 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:51.308248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:51.308076 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:51.342239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:51.335672 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:51.342239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:51.335724 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:51.729296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:51.726320 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:52.311239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.310593 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:52.311239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.310655 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:52.334241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.333844 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:52.334241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.333904 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:52.456041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.455290 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:52.508770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.508343 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:52.981248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:52.980290 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.035240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.035172 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:53.221241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.221026 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.282241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.280522 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.308241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.307589 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:53.308241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.307653 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:53.336239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.335607 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:53.336239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.335672 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:53.449241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.446356 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded" Nov 01 14:37:53.457237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.454296 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded" Nov 01 14:37:53.461239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.459812 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.461239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.460207 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:37:53.513244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.512310 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.513244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.512768 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:53.842240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.841934 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:53.842240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.842012 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.978428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.978387 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:37:53.978728 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.978682 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.997295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.987418 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:53.997295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:53.987760 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:37:54.454302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.332132 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:54.454302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.332189 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:54.454302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.400332 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:54.454302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.411366 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:37:54.462163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.461357 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:54.462163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.461436 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:54.464414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.464303 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 01 14:37:54.472416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.466179 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:54.472416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.466468 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:37:54.542543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.535774 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:54.619251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.604615 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:54.719573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.719484 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd"] Nov 01 14:37:54.774626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.773559 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8e4da22-9j7dd"] Nov 01 14:37:55.011300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:54.990388 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:55.059165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.052442 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:55.310021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.309584 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 01 14:37:55.310021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.309640 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 01 14:37:55.311644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.311609 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2db47471-1e23-498a-b586-d62da98f0679" path="/var/lib/kubelet/pods/2db47471-1e23-498a-b586-d62da98f0679/volumes" Nov 01 14:37:55.332822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.332773 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:55.333065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.333037 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:55.362067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.362025 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 14:37:55.458450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.458069 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:55.476328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.475588 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:37:55.476328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.476110 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:55.476328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.476231 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 01 14:37:55.498749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.498567 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 14:37:55.498914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.498797 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 14:37:55.499479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.499177 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 14:37:55.509454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:55.509438 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 14:37:56.142200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.142163 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:56.335241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.335132 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Nov 01 14:37:56.335241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.335209 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Nov 01 14:37:56.353252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.352695 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:56.685469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.684293 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:56.685469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.685316 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:56.686879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.686856 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:37:56.989687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.989617 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl"] Nov 01 14:37:56.995506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:56.995481 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.047586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.047556 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:37:57.047777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.047686 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:37:57.048467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.048445 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl"] Nov 01 14:37:57.062279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.062258 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.062397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.062382 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.062448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.062438 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5vp\" (UniqueName: \"kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.163422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.163391 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.163594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.163567 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.163669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.163630 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5vp\" (UniqueName: \"kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.164583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.164554 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.166042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.166022 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.220527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.220498 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5vp\" (UniqueName: \"kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp\") pod \"maas-api-revoke-sub-c98454f0-6j9bl\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.309763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.309735 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:37:57.338129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.338104 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:57.338520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.338499 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:57.339317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.339299 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:37:57.681997 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:37:57.681962 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311b1098_3806_4919_8ce0_4fc1df855b90.slice/crio-99b27dd425c7e42196ad9cb6849769c65b592aa3cd7c2089476cf59298671883 WatchSource:0}: Error finding container 99b27dd425c7e42196ad9cb6849769c65b592aa3cd7c2089476cf59298671883: Status 404 returned error can't find the container with id 99b27dd425c7e42196ad9cb6849769c65b592aa3cd7c2089476cf59298671883 Nov 01 14:37:57.698435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:57.690664 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl"] Nov 01 14:37:58.102671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:58.102644 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:37:58.556618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:58.556582 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" event={"ID":"311b1098-3806-4919-8ce0-4fc1df855b90","Type":"ContainerStarted","Data":"e831ceb6c3e3814dd8e14e6dea5204b79cda0106f109c0df11172070073cdd8c"} Nov 01 14:37:58.556774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:58.556640 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" event={"ID":"311b1098-3806-4919-8ce0-4fc1df855b90","Type":"ContainerStarted","Data":"99b27dd425c7e42196ad9cb6849769c65b592aa3cd7c2089476cf59298671883"} Nov 01 14:37:58.671934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:58.671890 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" podStartSLOduration=2.671875618 podStartE2EDuration="2.671875618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:37:58.669336856 +0000 UTC m=+1217.727488705" watchObservedRunningTime="2026-11-01 14:37:58.671875618 +0000 UTC m=+1217.730027522" Nov 01 14:37:59.576085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:59.576049 5473 generic.go:356] "Generic (PLEG): container finished" podID="311b1098-3806-4919-8ce0-4fc1df855b90" containerID="e831ceb6c3e3814dd8e14e6dea5204b79cda0106f109c0df11172070073cdd8c" exitCode=0 Nov 01 14:37:59.576488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:59.576114 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" event={"ID":"311b1098-3806-4919-8ce0-4fc1df855b90","Type":"ContainerDied","Data":"e831ceb6c3e3814dd8e14e6dea5204b79cda0106f109c0df11172070073cdd8c"} Nov 01 14:37:59.709842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:59.709805 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:37:59.721959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:37:59.721934 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:38:00.459683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.459646 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:00.597320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.597292 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 14:38:00.649081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.649046 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:00.743615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.743591 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:38:00.808154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.808129 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca\") pod \"311b1098-3806-4919-8ce0-4fc1df855b90\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " Nov 01 14:38:00.808311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.808201 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca\") pod \"311b1098-3806-4919-8ce0-4fc1df855b90\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " Nov 01 14:38:00.808371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.808358 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp\" (UniqueName: \"kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp\") pod \"311b1098-3806-4919-8ce0-4fc1df855b90\" (UID: \"311b1098-3806-4919-8ce0-4fc1df855b90\") " Nov 01 14:38:00.808583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.808545 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca" pod "311b1098-3806-4919-8ce0-4fc1df855b90" (UID: "311b1098-3806-4919-8ce0-4fc1df855b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:38:00.808822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.808797 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:00.810280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.810256 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca" pod "311b1098-3806-4919-8ce0-4fc1df855b90" (UID: "311b1098-3806-4919-8ce0-4fc1df855b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:38:00.810380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.810356 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp" pod "311b1098-3806-4919-8ce0-4fc1df855b90" (UID: "311b1098-3806-4919-8ce0-4fc1df855b90"). InnerVolumeSpecName "kube-api-access-2b5vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:00.846344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.846296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:00.910245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.910181 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311b1098-3806-4919-8ce0-4fc1df855b90-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:00.910245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:00.910241 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5vp\" (UniqueName: \"kubernetes.io/projected/311b1098-3806-4919-8ce0-4fc1df855b90-kube-api-access-2b5vp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:01.103265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.103236 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h"] Nov 01 14:38:01.103770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.103758 5473 cpu_manager.go:414] "containerMap: removing container" podUID="311b1098-3806-4919-8ce0-4fc1df855b90" containerName="revoke-keys" Nov 01 14:38:01.103816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.103772 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311b1098-3806-4919-8ce0-4fc1df855b90" containerName="revoke-keys" Nov 01 14:38:01.103912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.103902 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="311b1098-3806-4919-8ce0-4fc1df855b90" containerName="revoke-keys" Nov 01 14:38:01.107575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.107554 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.110416 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:38:01.110392 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:38:01.163087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.163060 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h"] Nov 01 14:38:01.213258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213187 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213396 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213442 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rngh4\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213495 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213568 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213594 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213690 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.213739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.213721 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315343 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315378 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315500 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315545 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rngh4\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315565 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315636 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315770 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315824 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315836 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315897 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.315985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.315916 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.316425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.316395 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.316519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.316493 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.316803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.316785 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.317994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.317971 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.318078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.318059 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.372430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.372377 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.372430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.372382 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rngh4\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.422148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.422127 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:01.593730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.593705 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" Nov 01 14:38:01.593910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.593743 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl" event={"ID":"311b1098-3806-4919-8ce0-4fc1df855b90","Type":"ContainerDied","Data":"99b27dd425c7e42196ad9cb6849769c65b592aa3cd7c2089476cf59298671883"} Nov 01 14:38:01.593910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.593790 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b27dd425c7e42196ad9cb6849769c65b592aa3cd7c2089476cf59298671883" Nov 01 14:38:01.853409 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:01.853381 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e80f9aa_1292_4bf7_9850_f3dd331e377d.slice/crio-f6a3d140e59fc8379fd93ec582dc19d53589278a4f38fee1f98eaba032214ccb WatchSource:0}: Error finding container f6a3d140e59fc8379fd93ec582dc19d53589278a4f38fee1f98eaba032214ccb: Status 404 returned error can't find the container with id f6a3d140e59fc8379fd93ec582dc19d53589278a4f38fee1f98eaba032214ccb Nov 01 14:38:01.856576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.856538 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:38:01.856697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.856650 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:38:01.856741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.856695 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:38:01.858965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:01.858941 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h"] Nov 01 14:38:02.604960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:02.604916 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" event={"ID":"5e80f9aa-1292-4bf7-9850-f3dd331e377d","Type":"ContainerStarted","Data":"c5c6fb20b55987e7afec9ab6f43710acf680de3a56ad6e98913df5471426e6eb"} Nov 01 14:38:02.605125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:02.605008 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" event={"ID":"5e80f9aa-1292-4bf7-9850-f3dd331e377d","Type":"ContainerStarted","Data":"f6a3d140e59fc8379fd93ec582dc19d53589278a4f38fee1f98eaba032214ccb"} Nov 01 14:38:02.792672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:02.792602 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podStartSLOduration=1.792583783 podStartE2EDuration="1.792583783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:02.770876231 +0000 UTC m=+1221.829028080" watchObservedRunningTime="2026-11-01 14:38:02.792583783 +0000 UTC m=+1221.850735637" Nov 01 14:38:02.865851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:02.864845 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:03.422320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:03.422282 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:03.424022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:03.423967 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 14:38:03.424138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:03.424046 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 14:38:03.511741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:03.511707 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:03.686205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:03.686118 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:04.087518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.084328 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f"] Nov 01 14:38:04.104268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.103318 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.146627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.126001 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f"] Nov 01 14:38:04.169257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.164020 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c9b0827c\"" Nov 01 14:38:04.169257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.164205 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c9b0827c-dockercfg-smdpw\"" Nov 01 14:38:04.176759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.176705 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.176897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.176785 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkjc\" (UniqueName: \"kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.278016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.277977 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.278249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.278025 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkjc\" (UniqueName: \"kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.278727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.278703 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.352618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.352545 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkjc\" (UniqueName: \"kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.422786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.422752 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 14:38:04.422944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.422802 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 14:38:04.464892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.464861 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:04.866738 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:04.866706 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2f6228_dbcb_4cce_bf56_4c1071bdb388.slice/crio-221cb7799ff6a8b3ed9ad25201f26958e5ad73e4dc9e47c92484b1368a1a421f WatchSource:0}: Error finding container 221cb7799ff6a8b3ed9ad25201f26958e5ad73e4dc9e47c92484b1368a1a421f: Status 404 returned error can't find the container with id 221cb7799ff6a8b3ed9ad25201f26958e5ad73e4dc9e47c92484b1368a1a421f Nov 01 14:38:04.871467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:04.871437 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f"] Nov 01 14:38:05.054574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.054535 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs"] Nov 01 14:38:05.058421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.058396 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684ba7ed-zcfhs"] Nov 01 14:38:05.084489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.084454 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cafbc8cc-a30d-485a-a77c-b2697c816620" path="/var/lib/kubelet/pods/cafbc8cc-a30d-485a-a77c-b2697c816620/volumes" Nov 01 14:38:05.423369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.423339 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 14:38:05.423757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.423388 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 14:38:05.686572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.686478 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" event={"ID":"9e2f6228-dbcb-4cce-bf56-4c1071bdb388","Type":"ContainerStarted","Data":"57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9"} Nov 01 14:38:05.686572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.686538 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" event={"ID":"9e2f6228-dbcb-4cce-bf56-4c1071bdb388","Type":"ContainerStarted","Data":"221cb7799ff6a8b3ed9ad25201f26958e5ad73e4dc9e47c92484b1368a1a421f"} Nov 01 14:38:05.686572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.686560 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:05.810623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:05.810577 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" podStartSLOduration=1.810565116 podStartE2EDuration="1.810565116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:05.805751523 +0000 UTC m=+1224.863903372" watchObservedRunningTime="2026-11-01 14:38:05.810565116 +0000 UTC m=+1224.868716981" Nov 01 14:38:06.422872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:06.422842 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 14:38:06.423042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:06.422900 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 14:38:07.422482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.422448 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 14:38:07.422873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.422505 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 14:38:07.503927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.503896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:07.694391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.694309 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g"] Nov 01 14:38:07.699210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.699191 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.751000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.750964 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g"] Nov 01 14:38:07.794976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.794944 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:38:07.794976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.794971 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:38:07.832502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.832475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.832662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.832539 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrrn\" (UniqueName: \"kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.832716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.832689 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.933954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.933917 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.934157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.934016 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.934157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.934087 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrrn\" (UniqueName: \"kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.934695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.934674 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.936249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.936199 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:07.990726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:07.990664 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrrn\" (UniqueName: \"kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn\") pod \"maas-api-revoke-sub-d0c6f8d3-q9s6g\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:08.013613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:08.013588 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:08.394882 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:08.394835 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d91415_945f_4ee2_b624_f52fcabef7b2.slice/crio-6bd00d910e3e1236b191f8e451b08f3ac9433e3437a61015be058a89ed5e0c5c WatchSource:0}: Error finding container 6bd00d910e3e1236b191f8e451b08f3ac9433e3437a61015be058a89ed5e0c5c: Status 404 returned error can't find the container with id 6bd00d910e3e1236b191f8e451b08f3ac9433e3437a61015be058a89ed5e0c5c Nov 01 14:38:08.398559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:08.398536 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g"] Nov 01 14:38:08.422630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:08.422590 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 01 14:38:08.422996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:08.422655 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 01 14:38:08.712503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:08.712408 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" event={"ID":"d3d91415-945f-4ee2-b624-f52fcabef7b2","Type":"ContainerStarted","Data":"6bd00d910e3e1236b191f8e451b08f3ac9433e3437a61015be058a89ed5e0c5c"} Nov 01 14:38:09.761291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:09.760801 5473 generic.go:356] "Generic (PLEG): container finished" podID="d3d91415-945f-4ee2-b624-f52fcabef7b2" containerID="643e28d582415e2ab41d49c7de07a680a018cfeea2bde7cff7a3879b88e59b9f" exitCode=0 Nov 01 14:38:09.761291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:09.761136 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" event={"ID":"d3d91415-945f-4ee2-b624-f52fcabef7b2","Type":"ContainerDied","Data":"643e28d582415e2ab41d49c7de07a680a018cfeea2bde7cff7a3879b88e59b9f"} Nov 01 14:38:10.464240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:10.463489 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:38:10.464240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:10.463552 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:38:11.478359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:11.477312 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:38:11.484161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:11.483415 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:38:11.554237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:11.539312 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:38:11.554237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:11.539374 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:38:11.554237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:11.541471 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 14:38:11.709290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:11.638523 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded" Nov 01 14:38:12.013700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.013673 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:12.130296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.127675 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn\" (UniqueName: \"kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn\") pod \"d3d91415-945f-4ee2-b624-f52fcabef7b2\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " Nov 01 14:38:12.130296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.127777 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca\") pod \"d3d91415-945f-4ee2-b624-f52fcabef7b2\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " Nov 01 14:38:12.130296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.127896 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca\") pod \"d3d91415-945f-4ee2-b624-f52fcabef7b2\" (UID: \"d3d91415-945f-4ee2-b624-f52fcabef7b2\") " Nov 01 14:38:12.130296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.129416 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca" pod "d3d91415-945f-4ee2-b624-f52fcabef7b2" (UID: "d3d91415-945f-4ee2-b624-f52fcabef7b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:38:12.173424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.173385 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn" pod "d3d91415-945f-4ee2-b624-f52fcabef7b2" (UID: "d3d91415-945f-4ee2-b624-f52fcabef7b2"). InnerVolumeSpecName "kube-api-access-sfrrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:12.231617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.231578 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:12.233664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.233639 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrrn\" (UniqueName: \"kubernetes.io/projected/d3d91415-945f-4ee2-b624-f52fcabef7b2-kube-api-access-sfrrn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:12.239359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.239332 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca" pod "d3d91415-945f-4ee2-b624-f52fcabef7b2" (UID: "d3d91415-945f-4ee2-b624-f52fcabef7b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:38:12.257592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.257562 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:12.258402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.258133 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:12.262816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.262795 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:38:12.335948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.335892 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d91415-945f-4ee2-b624-f52fcabef7b2-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:12.702087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.702059 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:12.985443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.985346 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" event={"ID":"d3d91415-945f-4ee2-b624-f52fcabef7b2","Type":"ContainerDied","Data":"6bd00d910e3e1236b191f8e451b08f3ac9433e3437a61015be058a89ed5e0c5c"} Nov 01 14:38:12.985443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.985414 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g" Nov 01 14:38:12.985660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:12.985444 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd00d910e3e1236b191f8e451b08f3ac9433e3437a61015be058a89ed5e0c5c" Nov 01 14:38:16.696453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:16.696426 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:38:17.098155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:17.098122 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:21.512333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:21.512299 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:22.049379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:22.049350 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn"] Nov 01 14:38:22.099733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:22.099701 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0ae16a-2r4bn"] Nov 01 14:38:23.081975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:23.081939 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111eb51e-10b3-4384-aa76-b5ae0e322842" path="/var/lib/kubelet/pods/111eb51e-10b3-4384-aa76-b5ae0e322842/volumes" Nov 01 14:38:24.060366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:24.060336 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:24.673468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:24.673426 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:24.857782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:24.857703 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:25.933980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:25.933946 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:26.968929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:26.968893 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr"] Nov 01 14:38:26.970297 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:26.970272 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d3d91415-945f-4ee2-b624-f52fcabef7b2" containerName="revoke-keys" Nov 01 14:38:26.970509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:26.970471 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d91415-945f-4ee2-b624-f52fcabef7b2" containerName="revoke-keys" Nov 01 14:38:26.970707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:26.970687 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d91415-945f-4ee2-b624-f52fcabef7b2" containerName="revoke-keys" Nov 01 14:38:26.976600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:26.976566 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.023976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.023922 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr"] Nov 01 14:38:27.034985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.034948 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kphsp\" (UniqueName: \"kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.035125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.035027 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.035306 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.035279 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.066369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.066338 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:38:27.066512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.066402 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:38:27.136770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.136737 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.136917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.136822 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kphsp\" (UniqueName: \"kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.136917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.136862 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.137564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.137543 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.139146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.139122 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.194332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.194302 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kphsp\" (UniqueName: \"kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp\") pod \"maas-api-revoke-sub-4542b32b-slcnr\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.299175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.299146 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:27.531586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.531557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:27.665383 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:27.665344 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecfc18f_cb9a_4bcf_a3e7_6cb5a463b770.slice/crio-977241f7b0ef14651359470377417e7be6e96844052afc0dcfe2dd3256ac3d73 WatchSource:0}: Error finding container 977241f7b0ef14651359470377417e7be6e96844052afc0dcfe2dd3256ac3d73: Status 404 returned error can't find the container with id 977241f7b0ef14651359470377417e7be6e96844052afc0dcfe2dd3256ac3d73 Nov 01 14:38:27.669751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:27.669728 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr"] Nov 01 14:38:28.104927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:28.104889 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" event={"ID":"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770","Type":"ContainerStarted","Data":"4a260ccb64d018c4069753e59ba55a647285b7b8cfff5de51073db34d2704284"} Nov 01 14:38:28.105328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:28.104943 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" event={"ID":"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770","Type":"ContainerStarted","Data":"977241f7b0ef14651359470377417e7be6e96844052afc0dcfe2dd3256ac3d73"} Nov 01 14:38:28.222065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:28.221972 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" podStartSLOduration=2.221957859 podStartE2EDuration="2.221957859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:28.217299603 +0000 UTC m=+1247.275451452" watchObservedRunningTime="2026-11-01 14:38:28.221957859 +0000 UTC m=+1247.280109707" Nov 01 14:38:28.308454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:28.308422 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc"] Nov 01 14:38:28.308690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:28.308662 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" containerID="cri-o://3507a8a791e01960554197d8aa9f7940b8ddc3a9f7ec3e92c058d78666a28bb0" gracePeriod=30 Nov 01 14:38:29.113883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:29.113843 5473 generic.go:356] "Generic (PLEG): container finished" podID="1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" containerID="4a260ccb64d018c4069753e59ba55a647285b7b8cfff5de51073db34d2704284" exitCode=0 Nov 01 14:38:29.113883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:29.113890 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" event={"ID":"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770","Type":"ContainerDied","Data":"4a260ccb64d018c4069753e59ba55a647285b7b8cfff5de51073db34d2704284"} Nov 01 14:38:29.584241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:29.584180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:29.756754 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:29.756717 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:29.917825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:29.917758 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:30.275014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.274989 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:30.373801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.373774 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca\") pod \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " Nov 01 14:38:30.373963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.373858 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca\") pod \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " Nov 01 14:38:30.374054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.374033 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp\" (UniqueName: \"kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp\") pod \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\" (UID: \"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770\") " Nov 01 14:38:30.374336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.374306 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca" pod "1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" (UID: "1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:38:30.374698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.374673 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:30.376078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.376045 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca" pod "1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" (UID: "1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:38:30.376179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.376121 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp" pod "1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" (UID: "1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770"). InnerVolumeSpecName "kube-api-access-kphsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:30.476391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.476318 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:30.476391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.476347 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kphsp\" (UniqueName: \"kubernetes.io/projected/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770-kube-api-access-kphsp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:30.504637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.504607 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:30.676822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.676785 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:30.841498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:30.841466 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:31.109180 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:38:31.109103 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:38:31.131644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:31.131544 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" Nov 01 14:38:31.131644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:31.131580 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr" event={"ID":"1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770","Type":"ContainerDied","Data":"977241f7b0ef14651359470377417e7be6e96844052afc0dcfe2dd3256ac3d73"} Nov 01 14:38:31.131644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:31.131623 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="977241f7b0ef14651359470377417e7be6e96844052afc0dcfe2dd3256ac3d73" Nov 01 14:38:34.022240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.021190 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:34.419078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.418995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:34.749667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.749583 5473 generic.go:356] "Generic (PLEG): container finished" podID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerID="3507a8a791e01960554197d8aa9f7940b8ddc3a9f7ec3e92c058d78666a28bb0" exitCode=0 Nov 01 14:38:34.750192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.750162 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" event={"ID":"ee55d16c-72ca-4f4b-8703-7e51dad943db","Type":"ContainerDied","Data":"3507a8a791e01960554197d8aa9f7940b8ddc3a9f7ec3e92c058d78666a28bb0"} Nov 01 14:38:34.750462 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.750441 5473 scope.go:122] "RemoveContainer" containerID="3861528aa02dc423c8cb6851f69393f215b7fac6256382207f5ef8a96fc3ed04" Nov 01 14:38:34.759048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.759028 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:38:34.891239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891056 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891143 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891203 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891555 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891272 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891555 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891319 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891555 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891438 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891555 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891478 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891555 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891552 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.891817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.891713 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket\") pod \"ee55d16c-72ca-4f4b-8703-7e51dad943db\" (UID: \"ee55d16c-72ca-4f4b-8703-7e51dad943db\") " Nov 01 14:38:34.905713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.905353 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:38:34.914764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.914322 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:38:34.927245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.915126 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:38:34.927982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.927947 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:38:34.928533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:34.928485 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:38:35.117248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.115481 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.117248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.115522 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.117248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.115539 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.117248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.115554 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee55d16c-72ca-4f4b-8703-7e51dad943db-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.117248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.115568 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.122255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.121721 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:38:35.122255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.121868 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "kube-api-access-srr65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:35.123100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.122695 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:38:35.125243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.124197 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c"] Nov 01 14:38:35.125243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.125145 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" Nov 01 14:38:35.125243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.125164 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" Nov 01 14:38:35.142439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.125194 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" containerName="revoke-keys" Nov 01 14:38:35.142439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.140127 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" containerName="revoke-keys" Nov 01 14:38:35.142439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.140228 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" Nov 01 14:38:35.142439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.140239 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" Nov 01 14:38:35.142439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.140420 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" containerName="revoke-keys" Nov 01 14:38:35.142439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.140437 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" Nov 01 14:38:35.157604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.157580 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.191255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.191209 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token" pod "ee55d16c-72ca-4f4b-8703-7e51dad943db" (UID: "ee55d16c-72ca-4f4b-8703-7e51dad943db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:35.220256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.219786 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.220256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.219912 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djc5v\" (UniqueName: \"kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v\") pod \"payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.220256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.220257 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.220563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.220280 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.220563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.220298 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srr65\" (UniqueName: \"kubernetes.io/projected/ee55d16c-72ca-4f4b-8703-7e51dad943db-kube-api-access-srr65\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.220563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.220315 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee55d16c-72ca-4f4b-8703-7e51dad943db-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:35.221496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.221471 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-84ded4-dockercfg-5t7gh\"" Nov 01 14:38:35.221618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.221563 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-84ded4\"" Nov 01 14:38:35.233235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.231321 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c"] Nov 01 14:38:35.322051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.322010 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.322329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.322303 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djc5v\" (UniqueName: \"kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v\") pod \"payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.323034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.322970 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.381432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.381382 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djc5v\" (UniqueName: \"kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v\") pod \"payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.508825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.508796 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:35.758428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.758399 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" Nov 01 14:38:35.758599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.758434 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc" event={"ID":"ee55d16c-72ca-4f4b-8703-7e51dad943db","Type":"ContainerDied","Data":"3cf5a40637b5b4cb3d301fedf8854af2c451d89e65bdf9ea57b134310053bfb6"} Nov 01 14:38:35.758599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.758484 5473 scope.go:122] "RemoveContainer" containerID="3507a8a791e01960554197d8aa9f7940b8ddc3a9f7ec3e92c058d78666a28bb0" Nov 01 14:38:35.889720 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:35.889681 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9cedbbf_2065_4809_86ed_817936c9cd50.slice/crio-648958da56d773cd4b59ed5035ac7d5114693010345ef290b7fef986a0ea45a0 WatchSource:0}: Error finding container 648958da56d773cd4b59ed5035ac7d5114693010345ef290b7fef986a0ea45a0: Status 404 returned error can't find the container with id 648958da56d773cd4b59ed5035ac7d5114693010345ef290b7fef986a0ea45a0 Nov 01 14:38:35.929550 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.929525 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c"] Nov 01 14:38:35.989335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.989310 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc"] Nov 01 14:38:35.993262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:35.993235 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-pfgxc"] Nov 01 14:38:36.686879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:36.686849 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:36.772362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:36.772325 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" event={"ID":"b9cedbbf-2065-4809-86ed-817936c9cd50","Type":"ContainerStarted","Data":"00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a"} Nov 01 14:38:36.772650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:36.772389 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" event={"ID":"b9cedbbf-2065-4809-86ed-817936c9cd50","Type":"ContainerStarted","Data":"648958da56d773cd4b59ed5035ac7d5114693010345ef290b7fef986a0ea45a0"} Nov 01 14:38:36.772650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:36.772487 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:36.885772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:36.885715 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" podStartSLOduration=2.885702809 podStartE2EDuration="2.885702809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:36.885379999 +0000 UTC m=+1255.943531848" watchObservedRunningTime="2026-11-01 14:38:36.885702809 +0000 UTC m=+1255.943854658" Nov 01 14:38:37.083421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:37.083382 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" path="/var/lib/kubelet/pods/ee55d16c-72ca-4f4b-8703-7e51dad943db/volumes" Nov 01 14:38:37.717888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:37.717856 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:37.883646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:37.883614 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:39.807664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:39.807631 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:41.557076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:41.557040 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:41.655209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:41.655177 5473 scope.go:122] "RemoveContainer" containerID="4ebcb3023ab5085a871cc38174ebe9f59c2952c4973101b71463635a8e6dfdb7" Nov 01 14:38:41.668758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:41.668736 5473 scope.go:122] "RemoveContainer" containerID="da1d1b4209fafd15cb968e13997b985dee4a134d976804a98de23e4144360404" Nov 01 14:38:41.678716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:41.678694 5473 scope.go:122] "RemoveContainer" containerID="9fadbe28eb20a0bfc74457cb48bcdc829f39930e26ea529f6ac72a02a4031002" Nov 01 14:38:41.688138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:41.688115 5473 scope.go:122] "RemoveContainer" containerID="e56eb7cb78c4ca4ac9f947408511f59e631248108d3db8c830726dc826370b79" Nov 01 14:38:43.328811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.328776 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm"] Nov 01 14:38:43.329693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.329670 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee55d16c-72ca-4f4b-8703-7e51dad943db" containerName="istio-proxy" Nov 01 14:38:43.335358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.335338 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.385011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.384982 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm"] Nov 01 14:38:43.418415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.418387 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wl9\" (UniqueName: \"kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.418552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.418519 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.418617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.418574 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.429912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.429882 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:38:43.430001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.429915 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:38:43.519482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.519455 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wl9\" (UniqueName: \"kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.519629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.519509 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.519629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.519535 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.520156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.520134 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.521854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.521834 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.576343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.576321 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wl9\" (UniqueName: \"kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9\") pod \"maas-api-revoke-sub-079f4d72-h8zdm\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:43.649066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:43.649011 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:44.016690 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:44.016662 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b52187_cf28_43ba_b3b5_a11f7f24b598.slice/crio-395605524c17d063854973ff4149a28b99cac3fd8c5cdcd44fc0db9d6bdd4060 WatchSource:0}: Error finding container 395605524c17d063854973ff4149a28b99cac3fd8c5cdcd44fc0db9d6bdd4060: Status 404 returned error can't find the container with id 395605524c17d063854973ff4149a28b99cac3fd8c5cdcd44fc0db9d6bdd4060 Nov 01 14:38:44.023832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:44.023805 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm"] Nov 01 14:38:44.838951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:44.838913 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" event={"ID":"61b52187-cf28-43ba-b3b5-a11f7f24b598","Type":"ContainerStarted","Data":"ebbd997afc6cb37c73e0870662099bf2e3680fe8bf9f47cb5fe8db3643c6a8c1"} Nov 01 14:38:44.839449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:44.839004 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" event={"ID":"61b52187-cf28-43ba-b3b5-a11f7f24b598","Type":"ContainerStarted","Data":"395605524c17d063854973ff4149a28b99cac3fd8c5cdcd44fc0db9d6bdd4060"} Nov 01 14:38:44.954302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:44.954256 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" podStartSLOduration=1.954242812 podStartE2EDuration="1.954242812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:44.953615519 +0000 UTC m=+1264.011767368" watchObservedRunningTime="2026-11-01 14:38:44.954242812 +0000 UTC m=+1264.012394661" Nov 01 14:38:45.847832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:45.847796 5473 generic.go:356] "Generic (PLEG): container finished" podID="61b52187-cf28-43ba-b3b5-a11f7f24b598" containerID="ebbd997afc6cb37c73e0870662099bf2e3680fe8bf9f47cb5fe8db3643c6a8c1" exitCode=0 Nov 01 14:38:45.848319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:45.847883 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" event={"ID":"61b52187-cf28-43ba-b3b5-a11f7f24b598","Type":"ContainerDied","Data":"ebbd997afc6cb37c73e0870662099bf2e3680fe8bf9f47cb5fe8db3643c6a8c1"} Nov 01 14:38:46.089713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.089682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:46.352954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.352915 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n"] Nov 01 14:38:46.357291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.357271 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.408749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.408722 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n"] Nov 01 14:38:46.452863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.452837 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.452996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.452877 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.453101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.453084 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ljz\" (UniqueName: \"kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.554528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.554499 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ljz\" (UniqueName: \"kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.554680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.554591 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.554680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.554619 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.555191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.555168 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.556830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.556812 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.611353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.611288 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ljz\" (UniqueName: \"kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz\") pod \"maas-api-revoke-sub-500188c1-8v22n\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.671283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.671256 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:46.997175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:46.997152 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:47.039254 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:47.039226 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231285ed_84db_4302_bc1c_fbd7761ff94c.slice/crio-5d80173358ecc005bd4229930d079fe66424c77ec40515f27b2d3b4baec5793f WatchSource:0}: Error finding container 5d80173358ecc005bd4229930d079fe66424c77ec40515f27b2d3b4baec5793f: Status 404 returned error can't find the container with id 5d80173358ecc005bd4229930d079fe66424c77ec40515f27b2d3b4baec5793f Nov 01 14:38:47.043529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.043508 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n"] Nov 01 14:38:47.163284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.163181 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9\" (UniqueName: \"kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9\") pod \"61b52187-cf28-43ba-b3b5-a11f7f24b598\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " Nov 01 14:38:47.163430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.163327 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca\") pod \"61b52187-cf28-43ba-b3b5-a11f7f24b598\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " Nov 01 14:38:47.163526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.163506 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca\") pod \"61b52187-cf28-43ba-b3b5-a11f7f24b598\" (UID: \"61b52187-cf28-43ba-b3b5-a11f7f24b598\") " Nov 01 14:38:47.163757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.163729 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca" pod "61b52187-cf28-43ba-b3b5-a11f7f24b598" (UID: "61b52187-cf28-43ba-b3b5-a11f7f24b598"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:38:47.164346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.164203 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:47.165759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.165729 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9" pod "61b52187-cf28-43ba-b3b5-a11f7f24b598" (UID: "61b52187-cf28-43ba-b3b5-a11f7f24b598"). InnerVolumeSpecName "kube-api-access-d6wl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:47.165759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.165754 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca" pod "61b52187-cf28-43ba-b3b5-a11f7f24b598" (UID: "61b52187-cf28-43ba-b3b5-a11f7f24b598"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:38:47.265086 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.265057 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b52187-cf28-43ba-b3b5-a11f7f24b598-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:47.265239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.265087 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6wl9\" (UniqueName: \"kubernetes.io/projected/61b52187-cf28-43ba-b3b5-a11f7f24b598-kube-api-access-d6wl9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:47.786256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.786230 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:38:47.864528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.864499 5473 generic.go:356] "Generic (PLEG): container finished" podID="231285ed-84db-4302-bc1c-fbd7761ff94c" containerID="a42be2ea8d111e605fccab92b560ef4dd7404fa00b98f26859f31724b789229a" exitCode=0 Nov 01 14:38:47.864672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.864583 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" event={"ID":"231285ed-84db-4302-bc1c-fbd7761ff94c","Type":"ContainerDied","Data":"a42be2ea8d111e605fccab92b560ef4dd7404fa00b98f26859f31724b789229a"} Nov 01 14:38:47.864672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.864637 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" event={"ID":"231285ed-84db-4302-bc1c-fbd7761ff94c","Type":"ContainerStarted","Data":"5d80173358ecc005bd4229930d079fe66424c77ec40515f27b2d3b4baec5793f"} Nov 01 14:38:47.866000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.865983 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" Nov 01 14:38:47.866108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.866027 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm" event={"ID":"61b52187-cf28-43ba-b3b5-a11f7f24b598","Type":"ContainerDied","Data":"395605524c17d063854973ff4149a28b99cac3fd8c5cdcd44fc0db9d6bdd4060"} Nov 01 14:38:47.866108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:47.866066 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395605524c17d063854973ff4149a28b99cac3fd8c5cdcd44fc0db9d6bdd4060" Nov 01 14:38:48.572315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:48.572288 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:48.864497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:48.864417 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:49.012334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.012306 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:49.185958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.185885 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca\") pod \"231285ed-84db-4302-bc1c-fbd7761ff94c\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " Nov 01 14:38:49.185958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.185933 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz\" (UniqueName: \"kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz\") pod \"231285ed-84db-4302-bc1c-fbd7761ff94c\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " Nov 01 14:38:49.186256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.186082 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca\") pod \"231285ed-84db-4302-bc1c-fbd7761ff94c\" (UID: \"231285ed-84db-4302-bc1c-fbd7761ff94c\") " Nov 01 14:38:49.186467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.186433 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca" pod "231285ed-84db-4302-bc1c-fbd7761ff94c" (UID: "231285ed-84db-4302-bc1c-fbd7761ff94c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:38:49.186665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.186646 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:49.187920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.187888 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca" pod "231285ed-84db-4302-bc1c-fbd7761ff94c" (UID: "231285ed-84db-4302-bc1c-fbd7761ff94c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:38:49.188073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.188050 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz" pod "231285ed-84db-4302-bc1c-fbd7761ff94c" (UID: "231285ed-84db-4302-bc1c-fbd7761ff94c"). InnerVolumeSpecName "kube-api-access-r8ljz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:38:49.287375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.287344 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231285ed-84db-4302-bc1c-fbd7761ff94c-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:49.287375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.287373 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ljz\" (UniqueName: \"kubernetes.io/projected/231285ed-84db-4302-bc1c-fbd7761ff94c-kube-api-access-r8ljz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:38:49.477875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.477791 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:49.881679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.881650 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" Nov 01 14:38:49.882091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.881681 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n" event={"ID":"231285ed-84db-4302-bc1c-fbd7761ff94c","Type":"ContainerDied","Data":"5d80173358ecc005bd4229930d079fe66424c77ec40515f27b2d3b4baec5793f"} Nov 01 14:38:49.882091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:49.881761 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d80173358ecc005bd4229930d079fe66424c77ec40515f27b2d3b4baec5793f" Nov 01 14:38:50.515758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:50.515654 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:55.478829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:55.478792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:55.710808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:55.710775 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:56.173789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.173759 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56"] Nov 01 14:38:56.174388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.174371 5473 cpu_manager.go:414] "containerMap: removing container" podUID="231285ed-84db-4302-bc1c-fbd7761ff94c" containerName="revoke-keys" Nov 01 14:38:56.174465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.174389 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231285ed-84db-4302-bc1c-fbd7761ff94c" containerName="revoke-keys" Nov 01 14:38:56.174465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.174430 5473 cpu_manager.go:414] "containerMap: removing container" podUID="61b52187-cf28-43ba-b3b5-a11f7f24b598" containerName="revoke-keys" Nov 01 14:38:56.174465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.174437 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b52187-cf28-43ba-b3b5-a11f7f24b598" containerName="revoke-keys" Nov 01 14:38:56.174572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.174562 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="231285ed-84db-4302-bc1c-fbd7761ff94c" containerName="revoke-keys" Nov 01 14:38:56.174606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.174577 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b52187-cf28-43ba-b3b5-a11f7f24b598" containerName="revoke-keys" Nov 01 14:38:56.178358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.178342 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202384 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202431 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202489 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202590 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202618 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202664 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202800 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202894 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.202972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.202924 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vjp\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.272346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.272312 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56"] Nov 01 14:38:56.282566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.282544 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:56.288693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.288637 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-466a6e-openshift-default-dockercfg-8ptdj\"" Nov 01 14:38:56.304376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304349 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304386 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304449 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304539 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304603 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304638 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vjp\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304713 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304767 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304816 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304902 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.304940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.304768 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.305417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.305393 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.305521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.305429 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.305586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.305529 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.307208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.307184 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.307316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.307287 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.371248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.369941 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vjp\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.371248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.370352 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token\") pod \"e2e-api-a-466a6e-openshift-default-6544855469-ggq56\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.492197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.492105 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:56.506981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.506952 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:56.917570 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:56.917534 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0bbcc5_0409_46ea_a9f5_c2e03e6bce2b.slice/crio-dbdce1be930970368a77acf17055f9b9952f9557c58a706c4225adec6fdb125d WatchSource:0}: Error finding container dbdce1be930970368a77acf17055f9b9952f9557c58a706c4225adec6fdb125d: Status 404 returned error can't find the container with id dbdce1be930970368a77acf17055f9b9952f9557c58a706c4225adec6fdb125d Nov 01 14:38:56.920026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.919970 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:38:56.920121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.920070 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:38:56.920121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.920113 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:38:56.922851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.922831 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56"] Nov 01 14:38:56.939605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:56.939575 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" event={"ID":"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b","Type":"ContainerStarted","Data":"dbdce1be930970368a77acf17055f9b9952f9557c58a706c4225adec6fdb125d"} Nov 01 14:38:57.212966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:57.212877 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:57.374844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:57.374811 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:57.948428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:57.948395 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" event={"ID":"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b","Type":"ContainerStarted","Data":"4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05"} Nov 01 14:38:58.115463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.115414 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podStartSLOduration=2.115400148 podStartE2EDuration="2.115400148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:38:58.114257105 +0000 UTC m=+1277.172408950" watchObservedRunningTime="2026-11-01 14:38:58.115400148 +0000 UTC m=+1277.173551993" Nov 01 14:38:58.492849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.492813 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:58.498667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.498638 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:58.501260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.501203 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts"] Nov 01 14:38:58.505361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.505342 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.535982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.535945 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.536120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.536025 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.536191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.536130 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.536191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.536171 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.536347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.536284 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9s7s\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.536347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.536330 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.536464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.536405 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.557786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.557758 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 14:38:58.557786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.557775 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 14:38:58.557911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.557788 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 14:38:58.557911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.557864 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lmbst\"" Nov 01 14:38:58.607938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.607908 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts"] Nov 01 14:38:58.637773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.637723 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.639007 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.639053 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.639084 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.639141 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9s7s\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.639184 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.639252 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.639725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.638929 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.645136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.644809 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.645136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.645095 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.645371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.645141 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.645371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.645304 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.698358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.698329 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.698706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.698675 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9s7s\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lzcts\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.819438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.819408 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:38:58.853489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.853457 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:58.958572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.958539 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:58.959367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:58.959341 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:38:59.040661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:59.040635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:38:59.279720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:59.279685 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts"] Nov 01 14:38:59.285527 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:38:59.285489 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a13785_e4fa_4d14_b930_7a447344288a.slice/crio-92f0d9a996b310bc8fa13eb754c51479d70910f5c34616bd65049c158aa54fd5 WatchSource:0}: Error finding container 92f0d9a996b310bc8fa13eb754c51479d70910f5c34616bd65049c158aa54fd5: Status 404 returned error can't find the container with id 92f0d9a996b310bc8fa13eb754c51479d70910f5c34616bd65049c158aa54fd5 Nov 01 14:38:59.291662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:59.289416 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 14:38:59.967065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:38:59.967033 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" event={"ID":"56a13785-e4fa-4d14-b930-7a447344288a","Type":"ContainerStarted","Data":"92f0d9a996b310bc8fa13eb754c51479d70910f5c34616bd65049c158aa54fd5"} Nov 01 14:39:00.173887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.173855 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc"] Nov 01 14:39:00.179482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.179460 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.230933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.230869 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc"] Nov 01 14:39:00.262118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.262091 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpr9v\" (UniqueName: \"kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.262279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.262151 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.262368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.262355 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.275459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.275434 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:39:00.275580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.275530 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:39:00.366240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.366056 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.366240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.366193 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.366515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.366400 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpr9v\" (UniqueName: \"kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.388238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.379610 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.388238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.380321 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.446239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.445953 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpr9v\" (UniqueName: \"kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v\") pod \"maas-api-revoke-sub-5801d564-n4crc\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.494225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.494134 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:00.669955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.669922 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:00.863539 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:39:00.863508 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfa8616_7302_4634_9fb4_0809868ffff6.slice/crio-d452b878c26617632f725cf6d58d21528d89a1c140449725d3e162539995263f WatchSource:0}: Error finding container d452b878c26617632f725cf6d58d21528d89a1c140449725d3e162539995263f: Status 404 returned error can't find the container with id d452b878c26617632f725cf6d58d21528d89a1c140449725d3e162539995263f Nov 01 14:39:00.868394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.868373 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc"] Nov 01 14:39:00.985605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:00.985572 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" event={"ID":"4dfa8616-7302-4634-9fb4-0809868ffff6","Type":"ContainerStarted","Data":"d452b878c26617632f725cf6d58d21528d89a1c140449725d3e162539995263f"} Nov 01 14:39:01.021662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:01.021626 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:01.108723 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:01.108690 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:39:01.199713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:01.199670 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:01.357837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:01.357790 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:01.997040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:01.996963 5473 generic.go:356] "Generic (PLEG): container finished" podID="4dfa8616-7302-4634-9fb4-0809868ffff6" containerID="8fe81f0a2afd3bb4249cd0d951ebdd91165b8330957540bc6cc547fc0f215ce2" exitCode=0 Nov 01 14:39:01.997040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:01.997016 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" event={"ID":"4dfa8616-7302-4634-9fb4-0809868ffff6","Type":"ContainerDied","Data":"8fe81f0a2afd3bb4249cd0d951ebdd91165b8330957540bc6cc547fc0f215ce2"} Nov 01 14:39:03.184443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.184417 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:03.230611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.230581 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:39:03.230701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.230644 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:39:03.314461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.314428 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v\" (UniqueName: \"kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v\") pod \"4dfa8616-7302-4634-9fb4-0809868ffff6\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " Nov 01 14:39:03.314654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.314629 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca\") pod \"4dfa8616-7302-4634-9fb4-0809868ffff6\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " Nov 01 14:39:03.314822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.314802 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca\") pod \"4dfa8616-7302-4634-9fb4-0809868ffff6\" (UID: \"4dfa8616-7302-4634-9fb4-0809868ffff6\") " Nov 01 14:39:03.315088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.315060 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca" pod "4dfa8616-7302-4634-9fb4-0809868ffff6" (UID: "4dfa8616-7302-4634-9fb4-0809868ffff6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:39:03.315485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.315451 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:03.316844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.316800 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v" pod "4dfa8616-7302-4634-9fb4-0809868ffff6" (UID: "4dfa8616-7302-4634-9fb4-0809868ffff6"). InnerVolumeSpecName "kube-api-access-hpr9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:39:03.319593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.319573 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca" pod "4dfa8616-7302-4634-9fb4-0809868ffff6" (UID: "4dfa8616-7302-4634-9fb4-0809868ffff6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:39:03.416435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.416402 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfa8616-7302-4634-9fb4-0809868ffff6-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:03.416435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:03.416430 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpr9v\" (UniqueName: \"kubernetes.io/projected/4dfa8616-7302-4634-9fb4-0809868ffff6-kube-api-access-hpr9v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:04.016105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.016062 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" event={"ID":"56a13785-e4fa-4d14-b930-7a447344288a","Type":"ContainerStarted","Data":"10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9"} Nov 01 14:39:04.016326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.016128 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:39:04.017524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.017501 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" Nov 01 14:39:04.017637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.017548 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc" event={"ID":"4dfa8616-7302-4634-9fb4-0809868ffff6","Type":"ContainerDied","Data":"d452b878c26617632f725cf6d58d21528d89a1c140449725d3e162539995263f"} Nov 01 14:39:04.017637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.017593 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d452b878c26617632f725cf6d58d21528d89a1c140449725d3e162539995263f" Nov 01 14:39:04.018245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.018196 5473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lzcts container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:39:04.018317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.018273 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" podUID="56a13785-e4fa-4d14-b930-7a447344288a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:39:04.224761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:04.224707 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" podStartSLOduration=2.286316686 podStartE2EDuration="6.224693918s" totalImagesPullingTime="3.938377232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:04.181056538 +0000 UTC m=+1283.239208388" watchObservedRunningTime="2026-11-01 14:39:04.224693918 +0000 UTC m=+1283.282845767" Nov 01 14:39:05.025704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:05.025674 5473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lzcts container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:39:05.025960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:05.025736 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" podUID="56a13785-e4fa-4d14-b930-7a447344288a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:39:05.454700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:05.454626 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:05.623811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:05.623780 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:05.797846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:05.797806 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:05.967208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:05.967174 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:06.123255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:06.123156 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:08.026249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:08.026207 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:39:10.079255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:10.076207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:11.122740 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:11.122710 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:12.790974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:12.790933 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:12.971934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:12.971903 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:14.142987 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.142949 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-04c6b8e2a5aa: secret "agc-cred-04c6b8e2a5aa" not found Nov 01 14:39:14.142987 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.142979 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-091b76ab0055: secret "agc-cred-091b76ab0055" not found Nov 01 14:39:14.142987 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.142985 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee13fb54a62a: secret "agc-cred-ee13fb54a62a" not found Nov 01 14:39:14.142987 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.142995 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc: [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:14.143732 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.143062 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials podName:d1ec8770-31d5-467e-95fb-2b7c950174d8 nodeName:}" failed. No retries permitted until 2026-11-01 14:39:14.643046212 +0000 UTC m=+1293.701198053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials") pod "payload-processing-external-model-6b765ff78c-ln8xc" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8") : [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:14.649345 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.649310 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-04c6b8e2a5aa: secret "agc-cred-04c6b8e2a5aa" not found Nov 01 14:39:14.649345 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.649338 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-091b76ab0055: secret "agc-cred-091b76ab0055" not found Nov 01 14:39:14.649345 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.649345 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee13fb54a62a: secret "agc-cred-ee13fb54a62a" not found Nov 01 14:39:14.649345 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.649355 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc: [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:14.649612 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:14.649420 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials podName:d1ec8770-31d5-467e-95fb-2b7c950174d8 nodeName:}" failed. No retries permitted until 2026-11-01 14:39:15.649405535 +0000 UTC m=+1294.707557377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials") pod "payload-processing-external-model-6b765ff78c-ln8xc" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8") : [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:14.792235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:14.792191 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc"] Nov 01 14:39:14.949809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:14.949725 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:15.670494 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:15.670454 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-04c6b8e2a5aa: secret "agc-cred-04c6b8e2a5aa" not found Nov 01 14:39:15.670494 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:15.670494 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-091b76ab0055: secret "agc-cred-091b76ab0055" not found Nov 01 14:39:15.671077 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:15.670507 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee13fb54a62a: secret "agc-cred-ee13fb54a62a" not found Nov 01 14:39:15.671077 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:15.670521 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc: [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:15.671077 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:15.670589 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials podName:d1ec8770-31d5-467e-95fb-2b7c950174d8 nodeName:}" failed. No retries permitted until 2026-11-01 14:39:17.670571214 +0000 UTC m=+1296.728723063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials") pod "payload-processing-external-model-6b765ff78c-ln8xc" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8") : [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:15.876966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:15.876931 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:16.037329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:16.037296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:16.647902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:16.647867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:17.700257 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:17.700206 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-04c6b8e2a5aa: secret "agc-cred-04c6b8e2a5aa" not found Nov 01 14:39:17.700257 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:17.700251 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-091b76ab0055: secret "agc-cred-091b76ab0055" not found Nov 01 14:39:17.700257 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:17.700260 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-ee13fb54a62a: secret "agc-cred-ee13fb54a62a" not found Nov 01 14:39:17.700746 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:17.700272 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc: [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:17.700746 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:17.700339 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials podName:d1ec8770-31d5-467e-95fb-2b7c950174d8 nodeName:}" failed. No retries permitted until 2026-11-01 14:39:21.700320344 +0000 UTC m=+1300.758472195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials") pod "payload-processing-external-model-6b765ff78c-ln8xc" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8") : [secret "agc-cred-04c6b8e2a5aa" not found, secret "agc-cred-091b76ab0055" not found, secret "agc-cred-ee13fb54a62a" not found] Nov 01 14:39:19.793099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:19.793049 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" podUID="d1ec8770-31d5-467e-95fb-2b7c950174d8" containerName="payload-processing" containerID="cri-o://ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc" gracePeriod=25 Nov 01 14:39:20.068521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.068500 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:39:20.140818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.140785 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz\") pod \"d1ec8770-31d5-467e-95fb-2b7c950174d8\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " Nov 01 14:39:20.140946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.140846 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume\") pod \"d1ec8770-31d5-467e-95fb-2b7c950174d8\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " Nov 01 14:39:20.140987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.140947 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume\") pod \"d1ec8770-31d5-467e-95fb-2b7c950174d8\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " Nov 01 14:39:20.141103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.141087 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials\") pod \"d1ec8770-31d5-467e-95fb-2b7c950174d8\" (UID: \"d1ec8770-31d5-467e-95fb-2b7c950174d8\") " Nov 01 14:39:20.141377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.141345 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume" pod "d1ec8770-31d5-467e-95fb-2b7c950174d8" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:39:20.141496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.141436 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume" pod "d1ec8770-31d5-467e-95fb-2b7c950174d8" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:39:20.141729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.141708 5473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-routing-overlay-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:20.141772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.141737 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec8770-31d5-467e-95fb-2b7c950174d8-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:20.143042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.143015 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz" pod "d1ec8770-31d5-467e-95fb-2b7c950174d8" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8"). InnerVolumeSpecName "kube-api-access-6smbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:39:20.143137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.143064 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials" pod "d1ec8770-31d5-467e-95fb-2b7c950174d8" (UID: "d1ec8770-31d5-467e-95fb-2b7c950174d8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:39:20.243787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.243748 5473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-provider-credentials\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:20.243787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.243787 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smbz\" (UniqueName: \"kubernetes.io/projected/d1ec8770-31d5-467e-95fb-2b7c950174d8-kube-api-access-6smbz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:39:20.274577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.274546 5473 generic.go:356] "Generic (PLEG): container finished" podID="d1ec8770-31d5-467e-95fb-2b7c950174d8" containerID="ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc" exitCode=0 Nov 01 14:39:20.274577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.274574 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" Nov 01 14:39:20.274744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.274615 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" event={"ID":"d1ec8770-31d5-467e-95fb-2b7c950174d8","Type":"ContainerDied","Data":"ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc"} Nov 01 14:39:20.274744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.274673 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc" event={"ID":"d1ec8770-31d5-467e-95fb-2b7c950174d8","Type":"ContainerDied","Data":"61774350ee85e6f64e4675eae8c31a313f6ef26683e015ad6cb7f1db5abbbfaf"} Nov 01 14:39:20.274744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.274701 5473 scope.go:122] "RemoveContainer" containerID="ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc" Nov 01 14:39:20.286651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.286631 5473 scope.go:122] "RemoveContainer" containerID="ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc" Nov 01 14:39:20.286908 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:20.286889 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc\": container with ID starting with ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc not found: ID does not exist" containerID="ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc" Nov 01 14:39:20.286969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.286916 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc"} err="failed to get container status \"ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc\": rpc error: code = NotFound desc = could not find container \"ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc\": container with ID starting with ab70e4e23158bcd12827fad14ffe3786418bb8301240ba6ff9941bbdcd9a78dc not found: ID does not exist" Nov 01 14:39:20.441244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.441148 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc"] Nov 01 14:39:20.445395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.445366 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b765ff78c-ln8xc"] Nov 01 14:39:20.710041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:20.709956 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:21.083779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:21.083747 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ec8770-31d5-467e-95fb-2b7c950174d8" path="/var/lib/kubelet/pods/d1ec8770-31d5-467e-95fb-2b7c950174d8/volumes" Nov 01 14:39:25.406131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:25.406094 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:29.952864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:29.952826 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:31.196871 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:39:31.196831 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:39:34.920535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:34.920476 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:38.636527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:38.636494 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:39.430525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:39.430486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:39.606662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:39.606620 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:40.317915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.317881 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf"] Nov 01 14:39:40.318504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.318489 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d1ec8770-31d5-467e-95fb-2b7c950174d8" containerName="payload-processing" Nov 01 14:39:40.318560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.318505 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ec8770-31d5-467e-95fb-2b7c950174d8" containerName="payload-processing" Nov 01 14:39:40.318560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.318522 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4dfa8616-7302-4634-9fb4-0809868ffff6" containerName="revoke-keys" Nov 01 14:39:40.318560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.318527 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfa8616-7302-4634-9fb4-0809868ffff6" containerName="revoke-keys" Nov 01 14:39:40.318668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.318625 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ec8770-31d5-467e-95fb-2b7c950174d8" containerName="payload-processing" Nov 01 14:39:40.318668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.318636 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfa8616-7302-4634-9fb4-0809868ffff6" containerName="revoke-keys" Nov 01 14:39:40.325748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.325728 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.377401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.377368 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf"] Nov 01 14:39:40.422156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.422115 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-84ded4-openshift-default-dockercfg-td8dn\"" Nov 01 14:39:40.462858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.462826 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.462914 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463031 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825ms\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463086 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463194 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463260 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463369 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463393 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.463428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.463412 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.564858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.564816 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.564903 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.564934 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.564961 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565128 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565185 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825ms\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565241 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565288 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565421 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565516 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565652 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.565849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.565697 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.566118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.566096 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.567510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.567489 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.567617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.567600 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.621661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.621589 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.622527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.622505 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-825ms\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms\") pod \"e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:40.639507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:40.639480 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:41.064306 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:39:41.064269 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde675b73_1072_4056_b1bb_755b6372827f.slice/crio-10774390d3d25639bfc06a45e04c5c6ba5c78b1321b1a5a10a98cd8dcaf4e68f WatchSource:0}: Error finding container 10774390d3d25639bfc06a45e04c5c6ba5c78b1321b1a5a10a98cd8dcaf4e68f: Status 404 returned error can't find the container with id 10774390d3d25639bfc06a45e04c5c6ba5c78b1321b1a5a10a98cd8dcaf4e68f Nov 01 14:39:41.066671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.066637 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:39:41.066780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.066703 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:39:41.066780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.066733 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:39:41.068595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.068566 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf"] Nov 01 14:39:41.485971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.485867 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" event={"ID":"de675b73-1072-4056-b1bb-755b6372827f","Type":"ContainerStarted","Data":"0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b"} Nov 01 14:39:41.485971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.485939 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" event={"ID":"de675b73-1072-4056-b1bb-755b6372827f","Type":"ContainerStarted","Data":"10774390d3d25639bfc06a45e04c5c6ba5c78b1321b1a5a10a98cd8dcaf4e68f"} Nov 01 14:39:41.639383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.633261 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podStartSLOduration=1.633237522 podStartE2EDuration="1.633237522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:39:41.627243898 +0000 UTC m=+1320.685395750" watchObservedRunningTime="2026-11-01 14:39:41.633237522 +0000 UTC m=+1320.691389372" Nov 01 14:39:41.639727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.639699 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:41.751349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.751243 5473 scope.go:122] "RemoveContainer" containerID="787bf1b26e416f21a4916be6b446bd2539c73bdacf913a24cab8ca5aea71aa3c" Nov 01 14:39:41.762705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.762682 5473 scope.go:122] "RemoveContainer" containerID="588f6cd30fdc601b436cb3a1ca7a1f3158bacb33d42bbd65300932db9eb1e5ae" Nov 01 14:39:41.773281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.773258 5473 scope.go:122] "RemoveContainer" containerID="7ad42e9f7ee82c73b5e2c961ecea43f92cf84dab2181cdd0486dd1c70ec8be07" Nov 01 14:39:41.783609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:41.783586 5473 scope.go:122] "RemoveContainer" containerID="0805cc368435ddd291dd367861172d0a96b5655f21e5813bed451ebdd87cc33d" Nov 01 14:39:42.647242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:42.644132 5473 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 14:39:42.647242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:42.644233 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 14:39:43.644602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:43.644564 5473 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 14:39:43.644910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:43.644877 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 14:39:44.645562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:44.645526 5473 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 14:39:44.646135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:44.646101 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 14:39:45.405133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:45.405095 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:45.647237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:45.646553 5473 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 14:39:45.647237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:45.646615 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 14:39:46.649538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:46.649499 5473 patch_prober.go:29] interesting pod/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 01 14:39:46.650031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:46.649555 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 01 14:39:47.644839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:47.644802 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:47.645267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:47.645241 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:47.645976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:47.645956 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:39:48.689461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:48.689426 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:49.811647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:49.811615 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:53.204398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:53.204345 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:56.033048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:56.033016 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:56.730073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:56.730035 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:57.697363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:57.697302 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:58.131818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:58.131782 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:39:58.303762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:39:58.303730 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:01.108071 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:40:01.108022 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:40:01.574015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:01.573984 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:01.750801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:01.750766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:01.940867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:01.940779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:02.900113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:02.900026 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:04.654853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:04.654821 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:04.832146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:04.832100 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:07.442526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.442488 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:07.743189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.743105 5473 generic.go:356] "Generic (PLEG): container finished" podID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerID="4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05" exitCode=0 Nov 01 14:40:07.743189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.743157 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" event={"ID":"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b","Type":"ContainerDied","Data":"4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05"} Nov 01 14:40:07.743586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.743571 5473 scope.go:122] "RemoveContainer" containerID="4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05" Nov 01 14:40:07.745076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.745050 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:07.745129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.745120 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:07.745164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:07.745153 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:08.758791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:08.758753 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" event={"ID":"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b","Type":"ContainerStarted","Data":"d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018"} Nov 01 14:40:09.493040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:09.493008 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:40:09.494014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:09.493989 5473 patch_prober.go:29] interesting pod/e2e-api-a-466a6e-openshift-default-6544855469-ggq56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 14:40:09.494141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:09.494052 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 14:40:10.492417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:10.492385 5473 patch_prober.go:29] interesting pod/e2e-api-a-466a6e-openshift-default-6544855469-ggq56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 14:40:10.492802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:10.492438 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 14:40:11.493394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:11.493362 5473 patch_prober.go:29] interesting pod/e2e-api-a-466a6e-openshift-default-6544855469-ggq56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 14:40:11.493757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:11.493410 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 14:40:11.844241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:11.841136 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:12.493278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:12.493243 5473 patch_prober.go:29] interesting pod/e2e-api-a-466a6e-openshift-default-6544855469-ggq56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 14:40:12.493459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:12.493300 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 14:40:13.492413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:13.492379 5473 patch_prober.go:29] interesting pod/e2e-api-a-466a6e-openshift-default-6544855469-ggq56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Nov 01 14:40:13.492586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:13.492431 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Nov 01 14:40:13.958632 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:13.958595 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:40:14.056648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:14.056612 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:14.497439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:14.497412 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:40:14.806362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:14.806335 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:40:16.286078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:16.286039 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:20.629483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:20.629403 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:25.193729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:25.193695 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:27.214404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.214372 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg"] Nov 01 14:40:27.219120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.219103 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.276689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.276669 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c9b0827c-openshift-default-dockercfg-4c75l\"" Nov 01 14:40:27.291342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291366 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9zm\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291442 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291510 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291591 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291644 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291681 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291731 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.291822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.291806 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.331529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.331498 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg"] Nov 01 14:40:27.393109 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393077 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393136 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9zm\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393168 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393272 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393378 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393430 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393488 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393519 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393526 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393581 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393598 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393865 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.393955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.393903 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.394160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.394142 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.396173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.396146 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.396299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.396171 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.407895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.407859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:27.451518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.451480 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.451746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.451723 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9zm\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm\") pod \"e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.532746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.532718 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:27.955323 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:40:27.955291 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0008a8_5b95_42e5_b0d3_956155648064.slice/crio-556073942b27db107e6f54b9078288ea14f0044d861b8f3c47c248d36cd83924 WatchSource:0}: Error finding container 556073942b27db107e6f54b9078288ea14f0044d861b8f3c47c248d36cd83924: Status 404 returned error can't find the container with id 556073942b27db107e6f54b9078288ea14f0044d861b8f3c47c248d36cd83924 Nov 01 14:40:27.957956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.957922 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:27.958075 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.958003 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:27.958075 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.958048 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:27.959823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:27.959799 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg"] Nov 01 14:40:28.092615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:28.092577 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:28.888101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:28.888073 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:28.949597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:28.938496 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" event={"ID":"ea0008a8-5b95-42e5-b0d3-956155648064","Type":"ContainerStarted","Data":"2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc"} Nov 01 14:40:28.949597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:28.938565 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" event={"ID":"ea0008a8-5b95-42e5-b0d3-956155648064","Type":"ContainerStarted","Data":"556073942b27db107e6f54b9078288ea14f0044d861b8f3c47c248d36cd83924"} Nov 01 14:40:29.071523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:29.071473 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podStartSLOduration=2.071457755 podStartE2EDuration="2.071457755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:29.067053956 +0000 UTC m=+1368.125205805" watchObservedRunningTime="2026-11-01 14:40:29.071457755 +0000 UTC m=+1368.129609605" Nov 01 14:40:29.169948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:29.169869 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:29.544240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:29.543520 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:29.545981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:29.545952 5473 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 14:40:29.546250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:29.546202 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 14:40:29.909202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:29.909127 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:30.544711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:30.544268 5473 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 14:40:30.544711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:30.544329 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 14:40:31.107998 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:40:31.107951 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:40:31.541269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:31.534554 5473 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 14:40:31.541269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:31.534612 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 14:40:32.543068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:32.543034 5473 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 14:40:32.543665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:32.543634 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 14:40:32.892705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:32.892591 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:33.088667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:33.088635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:33.272881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:33.272836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:33.542289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:33.540547 5473 patch_prober.go:29] interesting pod/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 01 14:40:33.542289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:33.540612 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 01 14:40:34.583170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:34.583141 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:34.584385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:34.584356 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:34.585334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:34.585302 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:40:34.585705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:34.585687 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:34.968434 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:34.968356 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:36.075116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.075083 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:36.264795 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.264766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:36.608802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.608757 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp"] Nov 01 14:40:36.614445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.614416 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.663107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.663077 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp"] Nov 01 14:40:36.707186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.707154 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-116e85\"" Nov 01 14:40:36.707646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.707618 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-116e85-dockercfg-7ltss\"" Nov 01 14:40:36.770357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.770317 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-116e85-5cd7498597-294sp\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.770609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.770586 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mls5g\" (UniqueName: \"kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g\") pod \"payload-processing-e2e-api-b-116e85-5cd7498597-294sp\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.871871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.871798 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-116e85-5cd7498597-294sp\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.872014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.871982 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mls5g\" (UniqueName: \"kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g\") pod \"payload-processing-e2e-api-b-116e85-5cd7498597-294sp\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.872518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.872496 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-116e85-5cd7498597-294sp\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.929501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.929473 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mls5g\" (UniqueName: \"kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g\") pod \"payload-processing-e2e-api-b-116e85-5cd7498597-294sp\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:36.936331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:36.936309 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:37.303123 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:40:37.303095 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0278c2_075b_4029_86e7_36d386f13fdb.slice/crio-cbea1ae78b4b454e63620a14460d81fc3463ac2d772c9e342b711496c630b241 WatchSource:0}: Error finding container cbea1ae78b4b454e63620a14460d81fc3463ac2d772c9e342b711496c630b241: Status 404 returned error can't find the container with id cbea1ae78b4b454e63620a14460d81fc3463ac2d772c9e342b711496c630b241 Nov 01 14:40:37.310377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:37.310348 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp"] Nov 01 14:40:38.045283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:38.045249 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" event={"ID":"7b0278c2-075b-4029-86e7-36d386f13fdb","Type":"ContainerStarted","Data":"d587063b2fb31d4da38cfee923baf78c271eb8b244f7c8c22e32f368d8260318"} Nov 01 14:40:38.045464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:38.045303 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" event={"ID":"7b0278c2-075b-4029-86e7-36d386f13fdb","Type":"ContainerStarted","Data":"cbea1ae78b4b454e63620a14460d81fc3463ac2d772c9e342b711496c630b241"} Nov 01 14:40:38.045464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:38.045331 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:38.164535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:38.164486 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" podStartSLOduration=2.164473138 podStartE2EDuration="2.164473138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:38.159919826 +0000 UTC m=+1377.218071675" watchObservedRunningTime="2026-11-01 14:40:38.164473138 +0000 UTC m=+1377.222624988" Nov 01 14:40:39.815258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:39.814797 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:40.859812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:40.859779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:41.454407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:41.453770 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:41.999041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:41.998958 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:44.263911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.263876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:44.348823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.348782 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744"] Nov 01 14:40:44.354007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.353983 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.409926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.409897 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744"] Nov 01 14:40:44.432705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.432663 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:44.453539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.453512 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536a87-openshift-default-dockercfg-588rv\"" Nov 01 14:40:44.471922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.471889 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.471936 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472060 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472123 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472317 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472377 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472434 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdsn\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472533 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.472590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.472557 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574122 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574173 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574199 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574274 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdsn\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574350 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574375 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574435 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574478 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574550 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574576 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.574804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574629 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.575021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.574980 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.575021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.575011 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.575148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.575048 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.576655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.576635 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.576831 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.576810 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.636003 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.635979 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.636113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.636047 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdsn\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.670169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.670146 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:44.718484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:44.718445 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:45.091558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:45.091529 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744"] Nov 01 14:40:45.122489 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:40:45.122461 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6bf672_93b8_4f7c_8531_e658951426dc.slice/crio-82316e327c6e36ddf6d613887c25a80a7ccb6e6f8f3190746f5526a3e3e258ee WatchSource:0}: Error finding container 82316e327c6e36ddf6d613887c25a80a7ccb6e6f8f3190746f5526a3e3e258ee: Status 404 returned error can't find the container with id 82316e327c6e36ddf6d613887c25a80a7ccb6e6f8f3190746f5526a3e3e258ee Nov 01 14:40:45.124843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:45.124814 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:45.124907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:45.124879 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:45.124945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:45.124910 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:40:46.110584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.110551 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" event={"ID":"5d6bf672-93b8-4f7c-8531-e658951426dc","Type":"ContainerStarted","Data":"7c225c26503d86f7f8a10f48053b116cce54b16c404d3ff74c904be4bf7e4c9e"} Nov 01 14:40:46.111119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.110604 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" event={"ID":"5d6bf672-93b8-4f7c-8531-e658951426dc","Type":"ContainerStarted","Data":"82316e327c6e36ddf6d613887c25a80a7ccb6e6f8f3190746f5526a3e3e258ee"} Nov 01 14:40:46.160248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.159974 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:46.319376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.319318 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podStartSLOduration=2.319303711 podStartE2EDuration="2.319303711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:46.281405638 +0000 UTC m=+1385.339557488" watchObservedRunningTime="2026-11-01 14:40:46.319303711 +0000 UTC m=+1385.377455561" Nov 01 14:40:46.361986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.361896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:46.672325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.671847 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:46.672325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.671970 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 01 14:40:46.672325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.672014 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 01 14:40:46.673460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:46.673378 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:47.670897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:47.670860 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 01 14:40:47.671415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:47.670916 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 01 14:40:48.671033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:48.670991 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 01 14:40:48.671463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:48.671057 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 01 14:40:49.057273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.057243 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:40:49.369767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.369689 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:49.575121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.575073 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:49.675291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.675133 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 01 14:40:49.675291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.675243 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 01 14:40:49.763238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.759410 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw"] Nov 01 14:40:49.771589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.771551 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:49.805421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.805381 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw"] Nov 01 14:40:49.816463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.816430 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsftw\" (UniqueName: \"kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw\") pod \"payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:49.816734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.816716 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:49.849417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.849377 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0c4976-dockercfg-9wlq9\"" Nov 01 14:40:49.853240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.849703 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0c4976\"" Nov 01 14:40:49.913614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.913582 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5"] Nov 01 14:40:49.920103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.918534 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsftw\" (UniqueName: \"kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw\") pod \"payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:49.920103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.918626 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:49.920103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.919968 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:49.921561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.921542 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:49.997154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:49.997098 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5"] Nov 01 14:40:50.086242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.027609 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhddw\" (UniqueName: \"kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw\") pod \"payload-processing-e2e-api-a-0c4976-6776786597-l6sb5\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.086781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.086741 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c4976-6776786597-l6sb5\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.095677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.095648 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsftw\" (UniqueName: \"kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw\") pod \"payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:50.100901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.100473 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:50.137054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.136960 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:50.189241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.189185 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c4976-6776786597-l6sb5\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.190037 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.189393 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhddw\" (UniqueName: \"kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw\") pod \"payload-processing-e2e-api-a-0c4976-6776786597-l6sb5\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.190037 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.189989 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0c4976-6776786597-l6sb5\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.256296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.256127 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhddw\" (UniqueName: \"kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw\") pod \"payload-processing-e2e-api-a-0c4976-6776786597-l6sb5\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.261165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.260940 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:50.351358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.351329 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:50.501825 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:40:50.501792 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff3f97f_5781_4714_8dde_a811c8d87223.slice/crio-3d999ac5d695165d51c4551a0ba753c06ceda060b16776e4e32b1a1e447aff5c WatchSource:0}: Error finding container 3d999ac5d695165d51c4551a0ba753c06ceda060b16776e4e32b1a1e447aff5c: Status 404 returned error can't find the container with id 3d999ac5d695165d51c4551a0ba753c06ceda060b16776e4e32b1a1e447aff5c Nov 01 14:40:50.512564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.512536 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw"] Nov 01 14:40:50.670283 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:40:50.670257 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4263e1a9_35d8_4a47_a735_041fb36ea5ea.slice/crio-0e11aadf9882aee6e941f7a8683780fbe0f8418a8cbe751345c0a484743e243c WatchSource:0}: Error finding container 0e11aadf9882aee6e941f7a8683780fbe0f8418a8cbe751345c0a484743e243c: Status 404 returned error can't find the container with id 0e11aadf9882aee6e941f7a8683780fbe0f8418a8cbe751345c0a484743e243c Nov 01 14:40:50.670541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.670523 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 01 14:40:50.670592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.670566 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 01 14:40:50.675059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:50.675033 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5"] Nov 01 14:40:51.175871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.175792 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" event={"ID":"4263e1a9-35d8-4a47-a735-041fb36ea5ea","Type":"ContainerStarted","Data":"d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f"} Nov 01 14:40:51.175871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.175842 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" event={"ID":"4263e1a9-35d8-4a47-a735-041fb36ea5ea","Type":"ContainerStarted","Data":"0e11aadf9882aee6e941f7a8683780fbe0f8418a8cbe751345c0a484743e243c"} Nov 01 14:40:51.176400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.176017 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:40:51.177661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.177635 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" event={"ID":"8ff3f97f-5781-4714-8dde-a811c8d87223","Type":"ContainerStarted","Data":"601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0"} Nov 01 14:40:51.177775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.177676 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" event={"ID":"8ff3f97f-5781-4714-8dde-a811c8d87223","Type":"ContainerStarted","Data":"3d999ac5d695165d51c4551a0ba753c06ceda060b16776e4e32b1a1e447aff5c"} Nov 01 14:40:51.177775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.177724 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:40:51.292702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.292654 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" podStartSLOduration=2.292642704 podStartE2EDuration="2.292642704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:51.29122917 +0000 UTC m=+1390.349381040" watchObservedRunningTime="2026-11-01 14:40:51.292642704 +0000 UTC m=+1390.350794554" Nov 01 14:40:51.412352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.412296 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" podStartSLOduration=2.412278556 podStartE2EDuration="2.412278556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:40:51.40603168 +0000 UTC m=+1390.464183531" watchObservedRunningTime="2026-11-01 14:40:51.412278556 +0000 UTC m=+1390.470430415" Nov 01 14:40:51.671439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.671403 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 01 14:40:51.671594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:51.671460 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 01 14:40:53.224697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:53.224654 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:53.225275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:53.225255 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:53.225995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:53.225973 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:40:54.504723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:54.504686 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:40:58.900533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:40:58.900503 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:01.106903 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:41:01.106864 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:41:02.189295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:02.189265 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:41:02.189689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:02.189341 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:41:03.413791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:03.413756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:07.820895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:07.820863 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:12.204858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:12.204827 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:16.707807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:16.707769 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:17.055030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.054997 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7"] Nov 01 14:41:17.104851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.104804 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb09241-9mdx7"] Nov 01 14:41:17.749680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.749650 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn"] Nov 01 14:41:17.755285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.755267 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.806381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.806352 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn"] Nov 01 14:41:17.850113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.850087 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:41:17.850275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.850167 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:41:17.867672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.867643 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqk4\" (UniqueName: \"kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.867862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.867843 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.867906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.867893 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.968917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.968885 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.969130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.968939 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.969130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.969050 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqk4\" (UniqueName: \"kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.969617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.969593 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:17.971496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:17.971468 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:18.029334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:18.029305 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqk4\" (UniqueName: \"kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4\") pod \"maas-api-revoke-sub-ff88d253-tqvzn\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:18.051428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:18.051391 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w"] Nov 01 14:41:18.056242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:18.056202 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b835e5ed-mgx8w"] Nov 01 14:41:18.069822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:18.069795 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:18.445085 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:41:18.445053 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4778b4_f0a1_41e8_a12a_13a4689d4583.slice/crio-cd0f4f4f356a05a40aedddbc5dc54f7dff18d8ca6ddae5808f2f943bdb560904 WatchSource:0}: Error finding container cd0f4f4f356a05a40aedddbc5dc54f7dff18d8ca6ddae5808f2f943bdb560904: Status 404 returned error can't find the container with id cd0f4f4f356a05a40aedddbc5dc54f7dff18d8ca6ddae5808f2f943bdb560904 Nov 01 14:41:18.449204 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:18.449179 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn"] Nov 01 14:41:19.083078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:19.083046 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af05ded-ba6c-425b-aa00-6328321ef265" path="/var/lib/kubelet/pods/0af05ded-ba6c-425b-aa00-6328321ef265/volumes" Nov 01 14:41:19.083564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:19.083549 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb" path="/var/lib/kubelet/pods/d493e9fa-b4b4-4c4e-a009-134fe6d8bcbb/volumes" Nov 01 14:41:19.378316 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:41:19.378252 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4778b4_f0a1_41e8_a12a_13a4689d4583.slice/crio-conmon-8a9c45f907484f2bd124f918f5db5bdd289bf51e1c7b6f6d0f731f2bb191b544.scope\": RecentStats: unable to find data in memory cache]" Nov 01 14:41:19.443467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:19.443431 5473 generic.go:356] "Generic (PLEG): container finished" podID="5f4778b4-f0a1-41e8-a12a-13a4689d4583" containerID="8a9c45f907484f2bd124f918f5db5bdd289bf51e1c7b6f6d0f731f2bb191b544" exitCode=0 Nov 01 14:41:19.443649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:19.443505 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" event={"ID":"5f4778b4-f0a1-41e8-a12a-13a4689d4583","Type":"ContainerDied","Data":"8a9c45f907484f2bd124f918f5db5bdd289bf51e1c7b6f6d0f731f2bb191b544"} Nov 01 14:41:19.443649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:19.443554 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" event={"ID":"5f4778b4-f0a1-41e8-a12a-13a4689d4583","Type":"ContainerStarted","Data":"cd0f4f4f356a05a40aedddbc5dc54f7dff18d8ca6ddae5808f2f943bdb560904"} Nov 01 14:41:20.611669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.611644 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:20.709689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.709618 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca\") pod \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " Nov 01 14:41:20.709835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.709794 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca\") pod \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " Nov 01 14:41:20.709913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.709897 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4\" (UniqueName: \"kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4\") pod \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\" (UID: \"5f4778b4-f0a1-41e8-a12a-13a4689d4583\") " Nov 01 14:41:20.709994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.709966 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca" pod "5f4778b4-f0a1-41e8-a12a-13a4689d4583" (UID: "5f4778b4-f0a1-41e8-a12a-13a4689d4583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:41:20.710446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.710427 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:20.711732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.711712 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca" pod "5f4778b4-f0a1-41e8-a12a-13a4689d4583" (UID: "5f4778b4-f0a1-41e8-a12a-13a4689d4583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:41:20.711881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.711861 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4" pod "5f4778b4-f0a1-41e8-a12a-13a4689d4583" (UID: "5f4778b4-f0a1-41e8-a12a-13a4689d4583"). InnerVolumeSpecName "kube-api-access-2dqk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:41:20.811902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.811865 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f4778b4-f0a1-41e8-a12a-13a4689d4583-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:20.811902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:20.811893 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dqk4\" (UniqueName: \"kubernetes.io/projected/5f4778b4-f0a1-41e8-a12a-13a4689d4583-kube-api-access-2dqk4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:21.154635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:21.154598 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:21.317360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:21.317320 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:21.460506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:21.460433 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" Nov 01 14:41:21.460506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:21.460449 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn" event={"ID":"5f4778b4-f0a1-41e8-a12a-13a4689d4583","Type":"ContainerDied","Data":"cd0f4f4f356a05a40aedddbc5dc54f7dff18d8ca6ddae5808f2f943bdb560904"} Nov 01 14:41:21.460687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:21.460522 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd0f4f4f356a05a40aedddbc5dc54f7dff18d8ca6ddae5808f2f943bdb560904" Nov 01 14:41:21.682410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:21.682371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:23.091673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.091639 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz"] Nov 01 14:41:23.092495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.092476 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5f4778b4-f0a1-41e8-a12a-13a4689d4583" containerName="revoke-keys" Nov 01 14:41:23.092495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.092495 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f4778b4-f0a1-41e8-a12a-13a4689d4583" containerName="revoke-keys" Nov 01 14:41:23.092647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.092606 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f4778b4-f0a1-41e8-a12a-13a4689d4583" containerName="revoke-keys" Nov 01 14:41:23.096592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.096571 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.152272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.149194 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:41:23.152272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.149762 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz"] Nov 01 14:41:23.194391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.194365 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:41:23.246024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.245989 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.246246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.246195 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czhmb\" (UniqueName: \"kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.246356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.246331 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.347366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.347281 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.347530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.347434 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czhmb\" (UniqueName: \"kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.347530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.347513 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.348102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.348079 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.349835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.349819 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.406877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.406842 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czhmb\" (UniqueName: \"kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb\") pod \"maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.410601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.410576 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:23.785034 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:41:23.784998 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd88a6ce_8a7a_4311_8438_fe6594f36016.slice/crio-5f0886eade1f29a2801fef91dfba1433e5fdfc451cd597cf9f183d034f842b1b WatchSource:0}: Error finding container 5f0886eade1f29a2801fef91dfba1433e5fdfc451cd597cf9f183d034f842b1b: Status 404 returned error can't find the container with id 5f0886eade1f29a2801fef91dfba1433e5fdfc451cd597cf9f183d034f842b1b Nov 01 14:41:23.790235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:23.790200 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz"] Nov 01 14:41:24.491452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:24.491368 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" event={"ID":"fd88a6ce-8a7a-4311-8438-fe6594f36016","Type":"ContainerStarted","Data":"cdff265425c0f79cde6dd22738dac3a1c2c1aa1ebbadf3f3fe899ba539fb6503"} Nov 01 14:41:24.491452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:24.491424 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" event={"ID":"fd88a6ce-8a7a-4311-8438-fe6594f36016","Type":"ContainerStarted","Data":"5f0886eade1f29a2801fef91dfba1433e5fdfc451cd597cf9f183d034f842b1b"} Nov 01 14:41:24.630577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:24.630527 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" podStartSLOduration=1.630512515 podStartE2EDuration="1.630512515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:41:24.624780133 +0000 UTC m=+1423.682931983" watchObservedRunningTime="2026-11-01 14:41:24.630512515 +0000 UTC m=+1423.688664365" Nov 01 14:41:25.500799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:25.500760 5473 generic.go:356] "Generic (PLEG): container finished" podID="fd88a6ce-8a7a-4311-8438-fe6594f36016" containerID="cdff265425c0f79cde6dd22738dac3a1c2c1aa1ebbadf3f3fe899ba539fb6503" exitCode=0 Nov 01 14:41:25.501331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:25.500805 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" event={"ID":"fd88a6ce-8a7a-4311-8438-fe6594f36016","Type":"ContainerDied","Data":"cdff265425c0f79cde6dd22738dac3a1c2c1aa1ebbadf3f3fe899ba539fb6503"} Nov 01 14:41:26.190147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.190098 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:26.664644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.664619 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:26.795902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.795869 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb\" (UniqueName: \"kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb\") pod \"fd88a6ce-8a7a-4311-8438-fe6594f36016\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " Nov 01 14:41:26.796080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.796050 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca\") pod \"fd88a6ce-8a7a-4311-8438-fe6594f36016\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " Nov 01 14:41:26.796146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.796129 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca\") pod \"fd88a6ce-8a7a-4311-8438-fe6594f36016\" (UID: \"fd88a6ce-8a7a-4311-8438-fe6594f36016\") " Nov 01 14:41:26.796476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.796446 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca" pod "fd88a6ce-8a7a-4311-8438-fe6594f36016" (UID: "fd88a6ce-8a7a-4311-8438-fe6594f36016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:41:26.796907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.796881 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:26.798374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.798344 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb" pod "fd88a6ce-8a7a-4311-8438-fe6594f36016" (UID: "fd88a6ce-8a7a-4311-8438-fe6594f36016"). InnerVolumeSpecName "kube-api-access-czhmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:41:26.798668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.798645 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca" pod "fd88a6ce-8a7a-4311-8438-fe6594f36016" (UID: "fd88a6ce-8a7a-4311-8438-fe6594f36016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:41:26.897741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.897709 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd88a6ce-8a7a-4311-8438-fe6594f36016-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:26.897741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:26.897740 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czhmb\" (UniqueName: \"kubernetes.io/projected/fd88a6ce-8a7a-4311-8438-fe6594f36016-kube-api-access-czhmb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:27.517801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:27.517763 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" Nov 01 14:41:27.517801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:27.517790 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz" event={"ID":"fd88a6ce-8a7a-4311-8438-fe6594f36016","Type":"ContainerDied","Data":"5f0886eade1f29a2801fef91dfba1433e5fdfc451cd597cf9f183d034f842b1b"} Nov 01 14:41:27.518008 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:27.517848 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0886eade1f29a2801fef91dfba1433e5fdfc451cd597cf9f183d034f842b1b" Nov 01 14:41:29.051119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:29.051085 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g"] Nov 01 14:41:29.101088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:29.101052 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-602a3a60-5m67g"] Nov 01 14:41:30.818033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:30.817998 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:31.083518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:31.083446 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5268ed-3ff3-44c1-a1ba-bee37afbcc68" path="/var/lib/kubelet/pods/3f5268ed-3ff3-44c1-a1ba-bee37afbcc68/volumes" Nov 01 14:41:31.108402 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:41:31.108372 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:41:33.049071 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:33.049041 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v"] Nov 01 14:41:33.098984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:33.098955 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-875a071b-mjq9v"] Nov 01 14:41:34.491248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:34.491198 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f"] Nov 01 14:41:34.861153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:34.861119 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:35.084484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:35.084453 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73567e3-6637-41b5-99c9-f78e60ea154f" path="/var/lib/kubelet/pods/e73567e3-6637-41b5-99c9-f78e60ea154f/volumes" Nov 01 14:41:36.858374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:36.858335 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:39.491954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:39.491884 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerName="payload-processing" containerID="cri-o://57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9" gracePeriod=25 Nov 01 14:41:40.359911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:40.359873 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:45.038563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:45.038516 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:45.060087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:45.060045 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll"] Nov 01 14:41:45.104250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:45.104192 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068d5f54-7lmll"] Nov 01 14:41:46.695456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:46.695418 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 14:41:47.083272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:47.083235 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b0ff2e-4cbb-4f91-a765-3d52ed71c654" path="/var/lib/kubelet/pods/45b0ff2e-4cbb-4f91-a765-3d52ed71c654/volumes" Nov 01 14:41:48.629602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:48.629563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:49.049610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:49.049578 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn"] Nov 01 14:41:49.099980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:49.099942 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b8a474d-q2kjn"] Nov 01 14:41:49.739034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:49.739002 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:50.466023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:50.465943 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:51.083509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:51.083478 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac984f1-d55b-49b3-ad34-ef434086d7b5" path="/var/lib/kubelet/pods/4ac984f1-d55b-49b3-ad34-ef434086d7b5/volumes" Nov 01 14:41:51.180684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:51.180649 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:54.113551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:54.113520 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:56.695100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:56.695056 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 14:41:57.227161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:57.227128 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:57.908472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:57.908441 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c"] Nov 01 14:41:58.177288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:58.177177 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:58.352913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:58.352842 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:58.802495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:58.802390 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:41:59.793632 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.793608 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:41:59.796505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.796480 5473 generic.go:356] "Generic (PLEG): container finished" podID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerID="57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9" exitCode=0 Nov 01 14:41:59.796652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.796512 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" Nov 01 14:41:59.796652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.796556 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" event={"ID":"9e2f6228-dbcb-4cce-bf56-4c1071bdb388","Type":"ContainerDied","Data":"57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9"} Nov 01 14:41:59.796652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.796608 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f" event={"ID":"9e2f6228-dbcb-4cce-bf56-4c1071bdb388","Type":"ContainerDied","Data":"221cb7799ff6a8b3ed9ad25201f26958e5ad73e4dc9e47c92484b1368a1a421f"} Nov 01 14:41:59.796652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.796637 5473 scope.go:122] "RemoveContainer" containerID="57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9" Nov 01 14:41:59.809828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.809808 5473 scope.go:122] "RemoveContainer" containerID="57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9" Nov 01 14:41:59.810097 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:41:59.810078 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9\": container with ID starting with 57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9 not found: ID does not exist" containerID="57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9" Nov 01 14:41:59.810170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.810109 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9"} err="failed to get container status \"57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9\": rpc error: code = NotFound desc = could not find container \"57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9\": container with ID starting with 57c8fe7cc4f93c3e9dd8aa3237866d0a525f616e22109206dfe21f97d74837a9 not found: ID does not exist" Nov 01 14:41:59.970324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.970293 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume\") pod \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " Nov 01 14:41:59.970506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.970352 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc\" (UniqueName: \"kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc\") pod \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\" (UID: \"9e2f6228-dbcb-4cce-bf56-4c1071bdb388\") " Nov 01 14:41:59.970736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.970710 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume" pod "9e2f6228-dbcb-4cce-bf56-4c1071bdb388" (UID: "9e2f6228-dbcb-4cce-bf56-4c1071bdb388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:41:59.970918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.970898 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:41:59.972412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:41:59.972384 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc" pod "9e2f6228-dbcb-4cce-bf56-4c1071bdb388" (UID: "9e2f6228-dbcb-4cce-bf56-4c1071bdb388"). InnerVolumeSpecName "kube-api-access-8tkjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:42:00.072027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:00.071979 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tkjc\" (UniqueName: \"kubernetes.io/projected/9e2f6228-dbcb-4cce-bf56-4c1071bdb388-kube-api-access-8tkjc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:00.219150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:00.219119 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f"] Nov 01 14:42:00.223175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:00.223152 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-jpc6f"] Nov 01 14:42:01.051754 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:01.051717 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh"] Nov 01 14:42:01.084065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:01.084030 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" path="/var/lib/kubelet/pods/9e2f6228-dbcb-4cce-bf56-4c1071bdb388/volumes" Nov 01 14:42:01.104069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:01.104040 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh"] Nov 01 14:42:01.112491 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:01.112459 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:42:01.142556 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:01.142516 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-b771e40a-7bkjh\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="22d76c85-4993-4586-b518-0be7e1a49232" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b771e40a-7bkjh" Nov 01 14:42:02.908931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:02.908824 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" podUID="b9cedbbf-2065-4809-86ed-817936c9cd50" containerName="payload-pre-processing" containerID="cri-o://00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a" gracePeriod=25 Nov 01 14:42:03.000931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.000903 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:03.096249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.089940 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d76c85-4993-4586-b518-0be7e1a49232" path="/var/lib/kubelet/pods/22d76c85-4993-4586-b518-0be7e1a49232/volumes" Nov 01 14:42:03.177643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.177549 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:42:03.223279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.223250 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume\") pod \"b9cedbbf-2065-4809-86ed-817936c9cd50\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " Nov 01 14:42:03.223437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.223342 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v\" (UniqueName: \"kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v\") pod \"b9cedbbf-2065-4809-86ed-817936c9cd50\" (UID: \"b9cedbbf-2065-4809-86ed-817936c9cd50\") " Nov 01 14:42:03.223719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.223692 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume" pod "b9cedbbf-2065-4809-86ed-817936c9cd50" (UID: "b9cedbbf-2065-4809-86ed-817936c9cd50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:42:03.224113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.224086 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9cedbbf-2065-4809-86ed-817936c9cd50-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:03.225543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.225516 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v" pod "b9cedbbf-2065-4809-86ed-817936c9cd50" (UID: "b9cedbbf-2065-4809-86ed-817936c9cd50"). InnerVolumeSpecName "kube-api-access-djc5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:42:03.325791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.325757 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djc5v\" (UniqueName: \"kubernetes.io/projected/b9cedbbf-2065-4809-86ed-817936c9cd50-kube-api-access-djc5v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:03.831776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.831738 5473 generic.go:356] "Generic (PLEG): container finished" podID="b9cedbbf-2065-4809-86ed-817936c9cd50" containerID="00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a" exitCode=0 Nov 01 14:42:03.831941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.831783 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" Nov 01 14:42:03.831941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.831795 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" event={"ID":"b9cedbbf-2065-4809-86ed-817936c9cd50","Type":"ContainerDied","Data":"00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a"} Nov 01 14:42:03.831941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.831844 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c" event={"ID":"b9cedbbf-2065-4809-86ed-817936c9cd50","Type":"ContainerDied","Data":"648958da56d773cd4b59ed5035ac7d5114693010345ef290b7fef986a0ea45a0"} Nov 01 14:42:03.831941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.831866 5473 scope.go:122] "RemoveContainer" containerID="00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a" Nov 01 14:42:03.843439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.843421 5473 scope.go:122] "RemoveContainer" containerID="00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a" Nov 01 14:42:03.843689 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:03.843671 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a\": container with ID starting with 00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a not found: ID does not exist" containerID="00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a" Nov 01 14:42:03.843762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.843702 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a"} err="failed to get container status \"00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a\": rpc error: code = NotFound desc = could not find container \"00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a\": container with ID starting with 00be784adf21b9a0d0a64738d2b68d411702824373e0ab8bc7ea7c9babc57d1a not found: ID does not exist" Nov 01 14:42:03.999054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:03.999022 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c"] Nov 01 14:42:04.003233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:04.003195 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84ded4-7995994ff5-pvx5c"] Nov 01 14:42:04.034691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:04.034666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:05.061625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:05.061592 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b"] Nov 01 14:42:05.082778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:05.082745 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9cedbbf-2065-4809-86ed-817936c9cd50" path="/var/lib/kubelet/pods/b9cedbbf-2065-4809-86ed-817936c9cd50/volumes" Nov 01 14:42:05.111652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:05.111624 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a4195b-s8l6b"] Nov 01 14:42:05.978031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:05.977996 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:06.443186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.443148 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h"] Nov 01 14:42:06.444169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444142 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerName="payload-processing" Nov 01 14:42:06.444360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444342 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerName="payload-processing" Nov 01 14:42:06.444486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444460 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fd88a6ce-8a7a-4311-8438-fe6594f36016" containerName="revoke-keys" Nov 01 14:42:06.444486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444482 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd88a6ce-8a7a-4311-8438-fe6594f36016" containerName="revoke-keys" Nov 01 14:42:06.444687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444526 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b9cedbbf-2065-4809-86ed-817936c9cd50" containerName="payload-pre-processing" Nov 01 14:42:06.444687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444535 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9cedbbf-2065-4809-86ed-817936c9cd50" containerName="payload-pre-processing" Nov 01 14:42:06.444792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444768 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9cedbbf-2065-4809-86ed-817936c9cd50" containerName="payload-pre-processing" Nov 01 14:42:06.444849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444795 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2f6228-dbcb-4cce-bf56-4c1071bdb388" containerName="payload-processing" Nov 01 14:42:06.444849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.444805 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd88a6ce-8a7a-4311-8438-fe6594f36016" containerName="revoke-keys" Nov 01 14:42:06.449945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.449926 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.512186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.510679 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-18a879-openshift-default-dockercfg-xlzkz\"" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.580914 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.580984 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581013 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581062 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581096 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhvn\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581136 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581180 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581201 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.581764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.581308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.593760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.593734 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h"] Nov 01 14:42:06.597359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.596792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:06.683906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.683870 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.683939 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.683999 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684029 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684081 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684116 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhvn\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684154 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684202 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684243 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.684773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.684751 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.686090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.686060 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.686557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.686537 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.686883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.686844 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.688618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.688597 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.698821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.698750 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.700853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.700833 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.771563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.771533 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.775473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.775442 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhvn\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn\") pod \"e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:06.979183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:06.979037 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:07.065693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.065663 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:07.082183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.082160 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f246f8-3cb2-446d-a362-c90b466d16d3" path="/var/lib/kubelet/pods/c5f246f8-3cb2-446d-a362-c90b466d16d3/volumes" Nov 01 14:42:07.488490 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:42:07.488462 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b77a03f_bf88_4ea5_abfa_ee50285637b1.slice/crio-e95794dddda5c6404a2d4d27a1b7fc3c9274cfd646ae47a389944c601f66d5f3 WatchSource:0}: Error finding container e95794dddda5c6404a2d4d27a1b7fc3c9274cfd646ae47a389944c601f66d5f3: Status 404 returned error can't find the container with id e95794dddda5c6404a2d4d27a1b7fc3c9274cfd646ae47a389944c601f66d5f3 Nov 01 14:42:07.491176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.491143 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:42:07.491275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.491230 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:42:07.491275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.491263 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:42:07.492622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.492600 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h"] Nov 01 14:42:07.870646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.870603 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" event={"ID":"0b77a03f-bf88-4ea5-abfa-ee50285637b1","Type":"ContainerStarted","Data":"7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba"} Nov 01 14:42:07.870815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.870665 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" event={"ID":"0b77a03f-bf88-4ea5-abfa-ee50285637b1","Type":"ContainerStarted","Data":"e95794dddda5c6404a2d4d27a1b7fc3c9274cfd646ae47a389944c601f66d5f3"} Nov 01 14:42:07.992294 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:07.992244 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podStartSLOduration=1.992227105 podStartE2EDuration="1.992227105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:42:07.987695864 +0000 UTC m=+1467.045847714" watchObservedRunningTime="2026-11-01 14:42:07.992227105 +0000 UTC m=+1467.050378952" Nov 01 14:42:08.066270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:08.066241 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:08.071630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:08.071610 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:08.756728 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:08.756688 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:08.878502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:08.878474 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:08.879464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:08.879446 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:42:11.405272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:11.405243 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:11.587872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:11.587839 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:11.843803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:11.843768 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:12.054417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:12.054384 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5"] Nov 01 14:42:12.104576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:12.104510 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f40d8592-bpcx5"] Nov 01 14:42:12.911924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:12.911894 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:13.086175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:13.086144 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b828607e-e25a-42dc-abd5-7376900bce3a" path="/var/lib/kubelet/pods/b828607e-e25a-42dc-abd5-7376900bce3a/volumes" Nov 01 14:42:14.791823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:14.791790 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:14.967601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:14.965653 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:16.050088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:16.050057 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6"] Nov 01 14:42:16.099738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:16.099709 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b5ed68a-xvfk6"] Nov 01 14:42:17.083479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:17.083447 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd8ce8f-6bcb-474b-91ea-4652f1959a30" path="/var/lib/kubelet/pods/2dd8ce8f-6bcb-474b-91ea-4652f1959a30/volumes" Nov 01 14:42:17.271660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:17.271622 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:17.438066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:17.437986 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:18.634802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:18.634765 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:18.999460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:18.999372 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz"] Nov 01 14:42:19.005466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.005442 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.069709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.069669 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf"] Nov 01 14:42:19.070081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.069939 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" containerID="cri-o://0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b" gracePeriod=30 Nov 01 14:42:19.071379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.071355 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:42:19.085074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.085042 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.085349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.085325 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.085523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.085507 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kqjm\" (UniqueName: \"kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.092968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.092936 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:19.098450 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.098406 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:42:19.109231 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.109191 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz"] Nov 01 14:42:19.199866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.199458 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.199866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.199589 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.199866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.199656 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kqjm\" (UniqueName: \"kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.201303 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.201275 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.212436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.212392 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.259140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.259056 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kqjm\" (UniqueName: \"kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm\") pod \"maas-api-revoke-sub-47e6a230-5k9pz\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.327809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.327773 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:19.714958 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:42:19.714921 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5134cc67_8517_441e_ad1b_313e231a372d.slice/crio-6b0ca024b130b4620a9452aa1c6d641af041f438bc3e59a8e0caa0331b0e6fcd WatchSource:0}: Error finding container 6b0ca024b130b4620a9452aa1c6d641af041f438bc3e59a8e0caa0331b0e6fcd: Status 404 returned error can't find the container with id 6b0ca024b130b4620a9452aa1c6d641af041f438bc3e59a8e0caa0331b0e6fcd Nov 01 14:42:19.719667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.719644 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz"] Nov 01 14:42:19.851602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.851569 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:19.982111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:19.982034 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" event={"ID":"5134cc67-8517-441e-ad1b-313e231a372d","Type":"ContainerStarted","Data":"6b0ca024b130b4620a9452aa1c6d641af041f438bc3e59a8e0caa0331b0e6fcd"} Nov 01 14:42:20.217120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:20.217092 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:20.993057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:20.992981 5473 generic.go:356] "Generic (PLEG): container finished" podID="5134cc67-8517-441e-ad1b-313e231a372d" containerID="ab67ec7cae69d36d0f681508691022b4fda4354ede1966148d5f9595305cc918" exitCode=0 Nov 01 14:42:20.993438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:20.993056 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" event={"ID":"5134cc67-8517-441e-ad1b-313e231a372d","Type":"ContainerDied","Data":"ab67ec7cae69d36d0f681508691022b4fda4354ede1966148d5f9595305cc918"} Nov 01 14:42:22.173615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.173591 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:22.342961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.342929 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca\") pod \"5134cc67-8517-441e-ad1b-313e231a372d\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " Nov 01 14:42:22.343135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.343064 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca\") pod \"5134cc67-8517-441e-ad1b-313e231a372d\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " Nov 01 14:42:22.343135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.343105 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm\" (UniqueName: \"kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm\") pod \"5134cc67-8517-441e-ad1b-313e231a372d\" (UID: \"5134cc67-8517-441e-ad1b-313e231a372d\") " Nov 01 14:42:22.343438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.343411 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca" pod "5134cc67-8517-441e-ad1b-313e231a372d" (UID: "5134cc67-8517-441e-ad1b-313e231a372d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:42:22.343683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.343668 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:22.345164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.345146 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm" pod "5134cc67-8517-441e-ad1b-313e231a372d" (UID: "5134cc67-8517-441e-ad1b-313e231a372d"). InnerVolumeSpecName "kube-api-access-7kqjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:42:22.345251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.345228 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca" pod "5134cc67-8517-441e-ad1b-313e231a372d" (UID: "5134cc67-8517-441e-ad1b-313e231a372d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:42:22.445081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.445049 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5134cc67-8517-441e-ad1b-313e231a372d-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:22.445081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:22.445079 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kqjm\" (UniqueName: \"kubernetes.io/projected/5134cc67-8517-441e-ad1b-313e231a372d-kube-api-access-7kqjm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:23.011525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:23.011494 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" Nov 01 14:42:23.011709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:23.011525 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz" event={"ID":"5134cc67-8517-441e-ad1b-313e231a372d","Type":"ContainerDied","Data":"6b0ca024b130b4620a9452aa1c6d641af041f438bc3e59a8e0caa0331b0e6fcd"} Nov 01 14:42:23.011709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:23.011580 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0ca024b130b4620a9452aa1c6d641af041f438bc3e59a8e0caa0331b0e6fcd" Nov 01 14:42:24.353617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.353591 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:42:24.374017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.373994 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374032 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374059 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374093 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374117 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374152 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374282 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:42:24.374496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374402 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374508 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:42:24.374565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374527 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:42:24.374565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374550 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374586 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:42:24.374716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374603 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket\") pod \"de675b73-1072-4056-b1bb-755b6372827f\" (UID: \"de675b73-1072-4056-b1bb-755b6372827f\") " Nov 01 14:42:24.374895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.374870 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:42:24.375178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.375159 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.375281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.375179 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.375281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.375189 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.375281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.375198 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de675b73-1072-4056-b1bb-755b6372827f-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.375281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.375207 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.376409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.376393 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:42:24.376710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.376689 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:42:24.376807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.376780 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:42:24.376807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.376783 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms" pod "de675b73-1072-4056-b1bb-755b6372827f" (UID: "de675b73-1072-4056-b1bb-755b6372827f"). InnerVolumeSpecName "kube-api-access-825ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:42:24.477159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.477126 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825ms\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-kube-api-access-825ms\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.477159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.477157 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de675b73-1072-4056-b1bb-755b6372827f-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.477384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.477171 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de675b73-1072-4056-b1bb-755b6372827f-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.477384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.477183 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de675b73-1072-4056-b1bb-755b6372827f-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:42:24.973357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:24.973299 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:25.034982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.034947 5473 generic.go:356] "Generic (PLEG): container finished" podID="de675b73-1072-4056-b1bb-755b6372827f" containerID="0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b" exitCode=0 Nov 01 14:42:25.035267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.035245 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" event={"ID":"de675b73-1072-4056-b1bb-755b6372827f","Type":"ContainerDied","Data":"0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b"} Nov 01 14:42:25.035425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.035409 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" event={"ID":"de675b73-1072-4056-b1bb-755b6372827f","Type":"ContainerDied","Data":"10774390d3d25639bfc06a45e04c5c6ba5c78b1321b1a5a10a98cd8dcaf4e68f"} Nov 01 14:42:25.035537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.035525 5473 scope.go:122] "RemoveContainer" containerID="0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b" Nov 01 14:42:25.035794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.035775 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf" Nov 01 14:42:25.048487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.048464 5473 scope.go:122] "RemoveContainer" containerID="0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b" Nov 01 14:42:25.048805 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:25.048787 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b\": container with ID starting with 0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b not found: ID does not exist" containerID="0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b" Nov 01 14:42:25.048854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.048815 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b"} err="failed to get container status \"0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b\": rpc error: code = NotFound desc = could not find container \"0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b\": container with ID starting with 0672999ea0a84ce5208a6c174177af573389cfc45bc6e125aad54ac351ba6d5b not found: ID does not exist" Nov 01 14:42:25.141151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.141112 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:25.204621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.204589 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf"] Nov 01 14:42:25.210009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.209985 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84ded4-openshift-default-d9cbb4847-qsqjf"] Nov 01 14:42:25.904700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:25.904670 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:26.809869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:26.809836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:27.082700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:27.082620 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de675b73-1072-4056-b1bb-755b6372827f" path="/var/lib/kubelet/pods/de675b73-1072-4056-b1bb-755b6372827f/volumes" Nov 01 14:42:27.623097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:27.623060 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:27.811391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:27.811362 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:29.280080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:29.280034 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:31.048072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:31.048036 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx"] Nov 01 14:42:31.098099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:31.098056 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx"] Nov 01 14:42:31.108156 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:31.108127 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:42:31.130455 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:31.130426 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-fb9db681-985rx\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="1d8e4c42-51e1-4b09-8988-928f889b17b8" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb9db681-985rx" Nov 01 14:42:31.312923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:31.312837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:31.671743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:31.671660 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:32.656382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:32.656350 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:33.082542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:33.082510 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8e4c42-51e1-4b09-8988-928f889b17b8" path="/var/lib/kubelet/pods/1d8e4c42-51e1-4b09-8988-928f889b17b8/volumes" Nov 01 14:42:33.089056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:33.089020 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:33.930243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:33.930181 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:34.304005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:34.303954 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:34.502544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:34.502483 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:34.789572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:34.789537 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:35.050722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:35.050638 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn"] Nov 01 14:42:35.100757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:35.100729 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0c7b31-dspgn"] Nov 01 14:42:36.105892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:36.105186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:36.306085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:36.306032 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:37.093375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:37.093340 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a250fb2-899b-4a51-a3f9-5d561451e541" path="/var/lib/kubelet/pods/7a250fb2-899b-4a51-a3f9-5d561451e541/volumes" Nov 01 14:42:38.198734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:38.198700 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:38.938443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:38.938410 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:39.151514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:39.151479 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:39.346373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:39.346343 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:41.049398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.049364 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg"] Nov 01 14:42:41.099159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.099131 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg"] Nov 01 14:42:41.131524 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:42:41.131493 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-be50e8e3-x5fcg\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="684c9c25-86c7-4307-8b1d-61b44a49c899" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be50e8e3-x5fcg" Nov 01 14:42:41.279549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.279479 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:41.898525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.898496 5473 scope.go:122] "RemoveContainer" containerID="1afa9db20cfbdc40fa8efe9e1b5450119e492c2d0db04828f8983e4b9543c87b" Nov 01 14:42:41.909395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.909372 5473 scope.go:122] "RemoveContainer" containerID="93b847ad2d3c8b03af26eecf41d8bc54b44b8272baebeed3cd3e1652078a9360" Nov 01 14:42:41.921425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.921398 5473 scope.go:122] "RemoveContainer" containerID="84ac1366631eb620ae8af09a4802354eb8c3f70fda8eb26c8dc7e963a39c076e" Nov 01 14:42:41.936487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:41.936421 5473 scope.go:122] "RemoveContainer" containerID="2cd1a3a324341d24cb7a24e1171b8b546a8a97b82883ff429c4276c18eb6a6e8" Nov 01 14:42:42.000572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:42.000543 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:42.194091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:42.193968 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:43.083364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:43.083331 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="684c9c25-86c7-4307-8b1d-61b44a49c899" path="/var/lib/kubelet/pods/684c9c25-86c7-4307-8b1d-61b44a49c899/volumes" Nov 01 14:42:43.651024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:43.651001 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:42:43.652865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:43.652843 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:42:44.080976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:44.080942 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:46.120656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:46.120628 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:48.048320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:48.048288 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt"] Nov 01 14:42:48.098135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:48.098107 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s"] Nov 01 14:42:48.101904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:48.101892 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6aa0eae4-2smrt"] Nov 01 14:42:48.105175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:48.105153 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-905256ee-szc6s"] Nov 01 14:42:49.082598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:49.082566 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271c7068-ade3-4b93-a0be-6eaac7715343" path="/var/lib/kubelet/pods/271c7068-ade3-4b93-a0be-6eaac7715343/volumes" Nov 01 14:42:49.083154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:49.083135 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a527c83-bab9-4e0c-894c-4fe97b8591c1" path="/var/lib/kubelet/pods/6a527c83-bab9-4e0c-894c-4fe97b8591c1/volumes" Nov 01 14:42:50.020375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:50.020338 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:50.835904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:50.835862 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:51.610276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:51.610242 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:56.262538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:56.262496 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:56.485803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:56.485768 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:58.721471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:58.721435 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:42:58.921587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:42:58.921555 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:00.224305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:00.224269 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:00.699326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:00.699282 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56"] Nov 01 14:43:00.699524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:00.699499 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" containerID="cri-o://d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018" gracePeriod=30 Nov 01 14:43:00.734758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:00.734729 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:01.050154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:01.050121 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl"] Nov 01 14:43:01.053881 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:01.053856 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c98454f0-6j9bl"] Nov 01 14:43:01.083791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:01.083765 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311b1098-3806-4919-8ce0-4fc1df855b90" path="/var/lib/kubelet/pods/311b1098-3806-4919-8ce0-4fc1df855b90/volumes" Nov 01 14:43:01.108876 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:01.108848 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:43:01.321764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:01.321672 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:01.526205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:01.526171 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:05.533359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:05.533325 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:06.032401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.032376 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:43:06.211786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.211706 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.211786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.211746 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.211786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.211781 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.211823 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.211889 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.211963 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.212077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.212129 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.212263 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert\") pod \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\" (UID: \"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b\") " Nov 01 14:43:06.212505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.212467 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:06.212777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.212751 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:06.212910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.212881 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:06.213095 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.213069 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:06.213247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.213209 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.213323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.213254 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.213323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.213270 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.213438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.213372 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:06.215602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.215571 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:43:06.215765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.215729 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:06.215880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.215786 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:06.215880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.215836 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp" pod "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" (UID: "0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b"). InnerVolumeSpecName "kube-api-access-r9vjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:06.314133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.314090 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.314133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.314129 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.314133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.314141 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vjp\" (UniqueName: \"kubernetes.io/projected/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-kube-api-access-r9vjp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.314389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.314150 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.314389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.314160 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.314389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.314169 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:06.401359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.399684 5473 generic.go:356] "Generic (PLEG): container finished" podID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerID="d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018" exitCode=0 Nov 01 14:43:06.401359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.399999 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" event={"ID":"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b","Type":"ContainerDied","Data":"d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018"} Nov 01 14:43:06.401359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.400061 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" event={"ID":"0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b","Type":"ContainerDied","Data":"dbdce1be930970368a77acf17055f9b9952f9557c58a706c4225adec6fdb125d"} Nov 01 14:43:06.401359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.400094 5473 scope.go:122] "RemoveContainer" containerID="d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018" Nov 01 14:43:06.401359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.400290 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.437018 5473 scope.go:122] "RemoveContainer" containerID="4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.450101 5473 scope.go:122] "RemoveContainer" containerID="d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:06.450431 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018\": container with ID starting with d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018 not found: ID does not exist" containerID="d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.450459 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018"} err="failed to get container status \"d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018\": rpc error: code = NotFound desc = could not find container \"d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018\": container with ID starting with d8239014889ccddf57c5b746a2ef7e221b5bf7d57edcfed2a2a8a53b65da9018 not found: ID does not exist" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.450477 5473 scope.go:122] "RemoveContainer" containerID="4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:06.450722 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05\": container with ID starting with 4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05 not found: ID does not exist" containerID="4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05" Nov 01 14:43:06.452475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.450750 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05"} err="failed to get container status \"4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05\": rpc error: code = NotFound desc = could not find container \"4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05\": container with ID starting with 4eccb29aeb971b0d04a645d4292bc974b5389846f1e5a6801bb10ec4b304ea05 not found: ID does not exist" Nov 01 14:43:06.579131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.579098 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56"] Nov 01 14:43:06.583290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:06.583266 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-466a6e-openshift-default-6544855469-ggq56"] Nov 01 14:43:07.083203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.083169 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" path="/var/lib/kubelet/pods/0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b/volumes" Nov 01 14:43:07.118007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.117982 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:07.315507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.315477 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:07.663545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.663515 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:07.698524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.698495 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz"] Nov 01 14:43:07.748012 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.747986 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c9b0827c-hxshz"] Nov 01 14:43:07.889536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:07.889501 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:08.311386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:08.311352 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp"] Nov 01 14:43:08.662412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:08.662326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:08.862891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:08.862855 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:09.086608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:09.083638 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd88a6ce-8a7a-4311-8438-fe6594f36016" path="/var/lib/kubelet/pods/fd88a6ce-8a7a-4311-8438-fe6594f36016/volumes" Nov 01 14:43:11.431581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:11.431551 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:12.410366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:12.410332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:13.053951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.053919 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g"] Nov 01 14:43:13.059562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.059537 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6f8d3-q9s6g"] Nov 01 14:43:13.082952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.082927 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d91415-945f-4ee2-b624-f52fcabef7b2" path="/var/lib/kubelet/pods/d3d91415-945f-4ee2-b624-f52fcabef7b2/volumes" Nov 01 14:43:13.312073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.311988 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" podUID="7b0278c2-075b-4029-86e7-36d386f13fdb" containerName="payload-processing" containerID="cri-o://d587063b2fb31d4da38cfee923baf78c271eb8b244f7c8c22e32f368d8260318" gracePeriod=25 Nov 01 14:43:13.410013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.409981 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:13.526861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.526829 5473 generic.go:356] "Generic (PLEG): container finished" podID="7b0278c2-075b-4029-86e7-36d386f13fdb" containerID="d587063b2fb31d4da38cfee923baf78c271eb8b244f7c8c22e32f368d8260318" exitCode=0 Nov 01 14:43:13.527028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.527015 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" event={"ID":"7b0278c2-075b-4029-86e7-36d386f13fdb","Type":"ContainerDied","Data":"d587063b2fb31d4da38cfee923baf78c271eb8b244f7c8c22e32f368d8260318"} Nov 01 14:43:13.614681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.614654 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:43:13.718900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.718864 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g\" (UniqueName: \"kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g\") pod \"7b0278c2-075b-4029-86e7-36d386f13fdb\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " Nov 01 14:43:13.719069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.719005 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume\") pod \"7b0278c2-075b-4029-86e7-36d386f13fdb\" (UID: \"7b0278c2-075b-4029-86e7-36d386f13fdb\") " Nov 01 14:43:13.719401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.719377 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume" pod "7b0278c2-075b-4029-86e7-36d386f13fdb" (UID: "7b0278c2-075b-4029-86e7-36d386f13fdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:13.719572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.719547 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0278c2-075b-4029-86e7-36d386f13fdb-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:13.721109 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.721084 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g" pod "7b0278c2-075b-4029-86e7-36d386f13fdb" (UID: "7b0278c2-075b-4029-86e7-36d386f13fdb"). InnerVolumeSpecName "kube-api-access-mls5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:13.820873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:13.820837 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mls5g\" (UniqueName: \"kubernetes.io/projected/7b0278c2-075b-4029-86e7-36d386f13fdb-kube-api-access-mls5g\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:14.536784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:14.536758 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" Nov 01 14:43:14.537194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:14.536781 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp" event={"ID":"7b0278c2-075b-4029-86e7-36d386f13fdb","Type":"ContainerDied","Data":"cbea1ae78b4b454e63620a14460d81fc3463ac2d772c9e342b711496c630b241"} Nov 01 14:43:14.537194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:14.536835 5473 scope.go:122] "RemoveContainer" containerID="d587063b2fb31d4da38cfee923baf78c271eb8b244f7c8c22e32f368d8260318" Nov 01 14:43:14.712497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:14.712464 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp"] Nov 01 14:43:14.716017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:14.715994 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-116e85-5cd7498597-294sp"] Nov 01 14:43:15.064089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:15.064055 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:15.082919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:15.082885 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0278c2-075b-4029-86e7-36d386f13fdb" path="/var/lib/kubelet/pods/7b0278c2-075b-4029-86e7-36d386f13fdb/volumes" Nov 01 14:43:15.508410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:15.508332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:17.556345 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:17.556296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:17.731369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:17.731331 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:18.030074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.030040 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62"] Nov 01 14:43:18.031248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031200 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" Nov 01 14:43:18.031388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031374 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" Nov 01 14:43:18.031550 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031539 5473 cpu_manager.go:414] "containerMap: removing container" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" Nov 01 14:43:18.031635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031624 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" Nov 01 14:43:18.031715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031705 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5134cc67-8517-441e-ad1b-313e231a372d" containerName="revoke-keys" Nov 01 14:43:18.031794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031783 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5134cc67-8517-441e-ad1b-313e231a372d" containerName="revoke-keys" Nov 01 14:43:18.031884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031874 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7b0278c2-075b-4029-86e7-36d386f13fdb" containerName="payload-processing" Nov 01 14:43:18.031980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.031968 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0278c2-075b-4029-86e7-36d386f13fdb" containerName="payload-processing" Nov 01 14:43:18.032259 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.032243 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" Nov 01 14:43:18.032401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.032387 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="de675b73-1072-4056-b1bb-755b6372827f" containerName="istio-proxy" Nov 01 14:43:18.032503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.032492 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5134cc67-8517-441e-ad1b-313e231a372d" containerName="revoke-keys" Nov 01 14:43:18.032594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.032583 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0278c2-075b-4029-86e7-36d386f13fdb" containerName="payload-processing" Nov 01 14:43:18.056719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.056691 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.101032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.100996 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62"] Nov 01 14:43:18.137239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.130242 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.137239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.130475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6pb\" (UniqueName: \"kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.150239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.139642 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c9b0827c-dockercfg-65qpg\"" Nov 01 14:43:18.233290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.147726 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c9b0827c\"" Nov 01 14:43:18.305329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.234470 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6pb\" (UniqueName: \"kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.305329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.266787 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.305329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.300672 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.305329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.305106 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6pb\" (UniqueName: \"kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb\") pod \"payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.313822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.313791 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq"] Nov 01 14:43:18.314668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.314640 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" Nov 01 14:43:18.314668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.314668 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" Nov 01 14:43:18.314884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.314868 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e0bbcc5-0409-46ea-a9f5-c2e03e6bce2b" containerName="istio-proxy" Nov 01 14:43:18.320945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.320924 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq"] Nov 01 14:43:18.321288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.321188 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.376129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.376108 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:18.471698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.470955 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.471698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.471246 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grk8p\" (UniqueName: \"kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.575558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.575301 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grk8p\" (UniqueName: \"kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.575558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.575375 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.576999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.576343 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.652475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.652406 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grk8p\" (UniqueName: \"kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p\") pod \"payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.680028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.680001 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:18.790589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:18.790405 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62"] Nov 01 14:43:19.089185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.089154 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:19.118943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.118921 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq"] Nov 01 14:43:19.652848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.652812 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" event={"ID":"042c5582-0f28-4bbd-8145-16489a946eb5","Type":"ContainerStarted","Data":"95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a"} Nov 01 14:43:19.653417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.652870 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" event={"ID":"042c5582-0f28-4bbd-8145-16489a946eb5","Type":"ContainerStarted","Data":"3ef282177c1f6b6041be55d228586b7de3e0dcb8fa8115e81676f4ff0f599114"} Nov 01 14:43:19.653417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.653190 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:19.654865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.654839 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" event={"ID":"40b711fd-a286-47cc-b661-090c7b7d0048","Type":"ContainerStarted","Data":"cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c"} Nov 01 14:43:19.654991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.654884 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" event={"ID":"40b711fd-a286-47cc-b661-090c7b7d0048","Type":"ContainerStarted","Data":"9549421bb26755d68b83e43128a89de5bcb138db0374d9aa338abda5f07bf608"} Nov 01 14:43:19.654991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.654912 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:19.890120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.890065 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" podStartSLOduration=1.890047194 podStartE2EDuration="1.890047194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:19.766515634 +0000 UTC m=+1538.824667484" watchObservedRunningTime="2026-11-01 14:43:19.890047194 +0000 UTC m=+1538.948199044" Nov 01 14:43:19.893255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:19.890657 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" podStartSLOduration=2.89064193 podStartE2EDuration="2.89064193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:19.886617989 +0000 UTC m=+1538.944769842" watchObservedRunningTime="2026-11-01 14:43:19.89064193 +0000 UTC m=+1538.948793781" Nov 01 14:43:21.294770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:21.294681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:21.475476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:21.475446 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:22.816582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:22.816551 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:23.294508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:23.294468 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:23.792076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:23.792041 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:24.194666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.194582 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh"] Nov 01 14:43:24.195140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.194807 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" containerID="cri-o://80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031" gracePeriod=30 Nov 01 14:43:24.220005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.219971 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6"] Nov 01 14:43:24.220175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.220154 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" containerID="cri-o://5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c" gracePeriod=30 Nov 01 14:43:24.244551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.244513 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x"] Nov 01 14:43:24.244835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.244731 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" containerID="cri-o://d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b" gracePeriod=30 Nov 01 14:43:24.251380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.251349 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9"] Nov 01 14:43:24.251633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.251602 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" containerID="cri-o://967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0" gracePeriod=30 Nov 01 14:43:24.920387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.920341 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:43:24.920603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:24.920413 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:43:25.702852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:25.702771 5473 generic.go:356] "Generic (PLEG): container finished" podID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerID="7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba" exitCode=0 Nov 01 14:43:25.702852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:25.702840 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" event={"ID":"0b77a03f-bf88-4ea5-abfa-ee50285637b1","Type":"ContainerDied","Data":"7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba"} Nov 01 14:43:25.703360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:25.703305 5473 scope.go:122] "RemoveContainer" containerID="7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba" Nov 01 14:43:25.704947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:25.704919 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:25.705032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:25.704981 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:25.705032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:25.705014 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:26.326800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:26.326708 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:43:26.326800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:26.326778 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:43:26.713242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:26.713135 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" event={"ID":"0b77a03f-bf88-4ea5-abfa-ee50285637b1","Type":"ContainerStarted","Data":"687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f"} Nov 01 14:43:26.952112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:26.952075 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5"] Nov 01 14:43:26.957367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:26.957332 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:26.988818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:26.988781 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5"] Nov 01 14:43:27.010227 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.010181 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:43:27.010227 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.010181 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:43:27.058548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.058509 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.058722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.058624 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ms76\" (UniqueName: \"kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.058722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.058678 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.066549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.066517 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:43:27.066742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.066720 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 01 14:43:27.066801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.066778 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 01 14:43:27.160696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.160650 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.160907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.160739 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ms76\" (UniqueName: \"kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.160907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.160801 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.161517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.161496 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.163107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.163083 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.218500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.218423 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ms76\" (UniqueName: \"kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76\") pod \"maas-api-revoke-sub-15da1799-c8jn5\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.271931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.271895 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:27.672080 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:43:27.672048 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ab0917_810c_476c_a535_9a031d74c629.slice/crio-536f801cc36e6bd0a47c7aac29479db0e836f1a8243d31ec736dc85493399930 WatchSource:0}: Error finding container 536f801cc36e6bd0a47c7aac29479db0e836f1a8243d31ec736dc85493399930: Status 404 returned error can't find the container with id 536f801cc36e6bd0a47c7aac29479db0e836f1a8243d31ec736dc85493399930 Nov 01 14:43:27.677522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.677493 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5"] Nov 01 14:43:27.722392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:27.722357 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" event={"ID":"c4ab0917-810c-476c-a535-9a031d74c629","Type":"ContainerStarted","Data":"536f801cc36e6bd0a47c7aac29479db0e836f1a8243d31ec736dc85493399930"} Nov 01 14:43:28.066277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.066247 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 01 14:43:28.066383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.066301 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 01 14:43:28.573769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.573702 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:43:28.573769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.573763 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:43:28.675678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.675290 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:28.772046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.772012 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" event={"ID":"c4ab0917-810c-476c-a535-9a031d74c629","Type":"ContainerStarted","Data":"c07bf4287c07cfb525f3e7bde64a37aba868ad5c63670a95a594407c81b22034"} Nov 01 14:43:28.955342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:28.955196 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:29.066465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.066429 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 01 14:43:29.066638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.066500 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 01 14:43:29.150622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.150586 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:29.280648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.280611 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 01 14:43:29.280807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.280678 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 01 14:43:29.617206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.616868 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:43:29.697103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.697070 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.698409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.698385 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.698995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.698975 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.699564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.699545 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.700708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.700687 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.700875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.700856 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.701277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.701256 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.701625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.701601 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.702828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.701703 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token\") pod \"37576851-40f6-42ab-bc50-be00a24bc5e1\" (UID: \"37576851-40f6-42ab-bc50-be00a24bc5e1\") " Nov 01 14:43:29.703364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.698275 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:29.703364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.698895 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.703364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.703200 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.703364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.703332 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.704077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.704050 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "kube-api-access-cgw5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.704880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.704326 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.704880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.704353 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.705042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.704932 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.705288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.705264 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:43:29.707577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.707552 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:43:29.708317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.708294 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo" pod "37576851-40f6-42ab-bc50-be00a24bc5e1" (UID: "37576851-40f6-42ab-bc50-be00a24bc5e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:43:29.711712 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.711693 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:43:29.783489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.783456 5473 generic.go:356] "Generic (PLEG): container finished" podID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerID="80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031" exitCode=0 Nov 01 14:43:29.783872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.783586 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" event={"ID":"7ce6f30b-c719-4010-a12b-6ef7de59940c","Type":"ContainerDied","Data":"80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031"} Nov 01 14:43:29.783872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.783629 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" event={"ID":"7ce6f30b-c719-4010-a12b-6ef7de59940c","Type":"ContainerDied","Data":"9533e1bd66a7e34b1547726996d080fbe4f6a5af413ce3fd8c94f5f433bd708a"} Nov 01 14:43:29.783872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.783657 5473 scope.go:122] "RemoveContainer" containerID="80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031" Nov 01 14:43:29.783872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.783803 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh" Nov 01 14:43:29.786467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.786428 5473 generic.go:356] "Generic (PLEG): container finished" podID="c4ab0917-810c-476c-a535-9a031d74c629" containerID="c07bf4287c07cfb525f3e7bde64a37aba868ad5c63670a95a594407c81b22034" exitCode=0 Nov 01 14:43:29.786726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.786679 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" event={"ID":"c4ab0917-810c-476c-a535-9a031d74c629","Type":"ContainerDied","Data":"c07bf4287c07cfb525f3e7bde64a37aba868ad5c63670a95a594407c81b22034"} Nov 01 14:43:29.788594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.788576 5473 generic.go:356] "Generic (PLEG): container finished" podID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerID="5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c" exitCode=0 Nov 01 14:43:29.788680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.788651 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" event={"ID":"007c0d84-c42e-4d3b-ad3c-cef5e43bf883","Type":"ContainerDied","Data":"5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c"} Nov 01 14:43:29.788730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.788689 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" event={"ID":"007c0d84-c42e-4d3b-ad3c-cef5e43bf883","Type":"ContainerDied","Data":"739a568f44cea6dbc8a70c3bc81bb871203fcfa9893b912c762b9e723de082eb"} Nov 01 14:43:29.788768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.788736 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6" Nov 01 14:43:29.790384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.790361 5473 generic.go:356] "Generic (PLEG): container finished" podID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerID="d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b" exitCode=0 Nov 01 14:43:29.790492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.790447 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" Nov 01 14:43:29.791126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.791104 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" event={"ID":"37576851-40f6-42ab-bc50-be00a24bc5e1","Type":"ContainerDied","Data":"d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b"} Nov 01 14:43:29.791299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.791148 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x" event={"ID":"37576851-40f6-42ab-bc50-be00a24bc5e1","Type":"ContainerDied","Data":"d34d4ea6a172eecc151e206fa78ad864c40f00ad1549f58496cef9becb1ecfa4"} Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804602 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804650 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804692 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804761 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804790 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804817 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804845 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804873 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804902 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804936 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.804967 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805021 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805074 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805106 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805164 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805235 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805269 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805313 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805345 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805377 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805457 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805489 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket\") pod \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\" (UID: \"007c0d84-c42e-4d3b-ad3c-cef5e43bf883\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805546 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.805574 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.806538 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.806749 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.806910 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs\") pod \"0ff736db-b409-4a27-b198-0108ac29adc5\" (UID: \"0ff736db-b409-4a27-b198-0108ac29adc5\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.806951 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs\") pod \"7ce6f30b-c719-4010-a12b-6ef7de59940c\" (UID: \"7ce6f30b-c719-4010-a12b-6ef7de59940c\") " Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.807481 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.807693 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.807884 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.807973 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.807999 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgw5l\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-kube-api-access-cgw5l\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808016 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808030 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808045 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808059 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808059 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808073 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37576851-40f6-42ab-bc50-be00a24bc5e1-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808087 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808101 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808116 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808131 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37576851-40f6-42ab-bc50-be00a24bc5e1-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.808243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808147 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37576851-40f6-42ab-bc50-be00a24bc5e1-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.808344 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.811459 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.812927 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813103 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ff736db-b409-4a27-b198-0108ac29adc5" containerID="967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0" exitCode=0 Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813151 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" event={"ID":"0ff736db-b409-4a27-b198-0108ac29adc5","Type":"ContainerDied","Data":"967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0"} Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813199 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" event={"ID":"0ff736db-b409-4a27-b198-0108ac29adc5","Type":"ContainerDied","Data":"e335492e781adb87219eb8d3db8ca304293aa3effffee0b493c5b9ff53c26111"} Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813318 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813371 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813623 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.813980 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.814189 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.814351 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.814459 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.814751 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.815242 5473 scope.go:122] "RemoveContainer" containerID="80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.814833 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.815023 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.816284 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "kube-api-access-r8747". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.816458 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:29.817393 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031\": container with ID starting with 80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031 not found: ID does not exist" containerID="80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.817418 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031"} err="failed to get container status \"80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031\": rpc error: code = NotFound desc = could not find container \"80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031\": container with ID starting with 80cc910aa7b9e541a95d24015d72978e12b06e45564756563b81b3d9a3869031 not found: ID does not exist" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.817433 5473 scope.go:122] "RemoveContainer" containerID="5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.817755 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.821600 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.818336 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "kube-api-access-mlch2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.859257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.856850 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token" pod "0ff736db-b409-4a27-b198-0108ac29adc5" (UID: "0ff736db-b409-4a27-b198-0108ac29adc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.859257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.856948 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.859257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.857681 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:29.859257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.858468 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "kube-api-access-hqm8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:29.860509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.860478 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo" pod "7ce6f30b-c719-4010-a12b-6ef7de59940c" (UID: "7ce6f30b-c719-4010-a12b-6ef7de59940c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:43:29.862541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.862504 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo" pod "007c0d84-c42e-4d3b-ad3c-cef5e43bf883" (UID: "007c0d84-c42e-4d3b-ad3c-cef5e43bf883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:43:29.868200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.867901 5473 scope.go:122] "RemoveContainer" containerID="5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c" Nov 01 14:43:29.868896 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:29.868493 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c\": container with ID starting with 5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c not found: ID does not exist" containerID="5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c" Nov 01 14:43:29.868896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.868532 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c"} err="failed to get container status \"5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c\": rpc error: code = NotFound desc = could not find container \"5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c\": container with ID starting with 5358e1499d31cd7b0171041b703900f37ce14a67b28440a9e5cf7d5452b75f4c not found: ID does not exist" Nov 01 14:43:29.868896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.868555 5473 scope.go:122] "RemoveContainer" containerID="d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b" Nov 01 14:43:29.884207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.884118 5473 scope.go:122] "RemoveContainer" containerID="d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b" Nov 01 14:43:29.884501 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:29.884451 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b\": container with ID starting with d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b not found: ID does not exist" containerID="d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b" Nov 01 14:43:29.884501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.884480 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b"} err="failed to get container status \"d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b\": rpc error: code = NotFound desc = could not find container \"d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b\": container with ID starting with d2013e69c2f1458cf82ac9832d458f74e40d5faa8f18b6fc516d4d8e97b3da6b not found: ID does not exist" Nov 01 14:43:29.884501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.884499 5473 scope.go:122] "RemoveContainer" containerID="967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0" Nov 01 14:43:29.896731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.896639 5473 scope.go:122] "RemoveContainer" containerID="967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0" Nov 01 14:43:29.897599 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:29.897473 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0\": container with ID starting with 967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0 not found: ID does not exist" containerID="967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0" Nov 01 14:43:29.897599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.897511 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0"} err="failed to get container status \"967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0\": rpc error: code = NotFound desc = could not find container \"967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0\": container with ID starting with 967332bd77b06aad22361142478c0d1f57abf099b48c484aba5642891df725d0 not found: ID does not exist" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909883 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909915 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909930 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909944 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909957 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff736db-b409-4a27-b198-0108ac29adc5-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909970 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909984 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.909997 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910010 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910024 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910039 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910052 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff736db-b409-4a27-b198-0108ac29adc5-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910065 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910078 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910091 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910104 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910117 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqm8r\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-kube-api-access-hqm8r\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910130 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910143 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007c0d84-c42e-4d3b-ad3c-cef5e43bf883-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910157 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8747\" (UniqueName: \"kubernetes.io/projected/0ff736db-b409-4a27-b198-0108ac29adc5-kube-api-access-r8747\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910171 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlch2\" (UniqueName: \"kubernetes.io/projected/7ce6f30b-c719-4010-a12b-6ef7de59940c-kube-api-access-mlch2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910184 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce6f30b-c719-4010-a12b-6ef7de59940c-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910198 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff736db-b409-4a27-b198-0108ac29adc5-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:29.910358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:29.910263 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce6f30b-c719-4010-a12b-6ef7de59940c-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:30.075070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.074913 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 01 14:43:30.075070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.075047 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 01 14:43:30.080941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.080915 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:30.218935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.218902 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca\") pod \"c4ab0917-810c-476c-a535-9a031d74c629\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " Nov 01 14:43:30.219127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.218961 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca\") pod \"c4ab0917-810c-476c-a535-9a031d74c629\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " Nov 01 14:43:30.219127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.219108 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76\" (UniqueName: \"kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76\") pod \"c4ab0917-810c-476c-a535-9a031d74c629\" (UID: \"c4ab0917-810c-476c-a535-9a031d74c629\") " Nov 01 14:43:30.219445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.219416 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca" pod "c4ab0917-810c-476c-a535-9a031d74c629" (UID: "c4ab0917-810c-476c-a535-9a031d74c629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:30.219820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.219792 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:30.221192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.221172 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca" pod "c4ab0917-810c-476c-a535-9a031d74c629" (UID: "c4ab0917-810c-476c-a535-9a031d74c629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:43:30.221287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.221205 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76" pod "c4ab0917-810c-476c-a535-9a031d74c629" (UID: "c4ab0917-810c-476c-a535-9a031d74c629"). InnerVolumeSpecName "kube-api-access-9ms76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:30.273460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.273429 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9"] Nov 01 14:43:30.279297 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.279273 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9"] Nov 01 14:43:30.321502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.321470 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4ab0917-810c-476c-a535-9a031d74c629-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:30.321502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.321497 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ms76\" (UniqueName: \"kubernetes.io/projected/c4ab0917-810c-476c-a535-9a031d74c629-kube-api-access-9ms76\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:30.386350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.386264 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh"] Nov 01 14:43:30.390121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.390089 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-fxlmh"] Nov 01 14:43:30.505836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.505802 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x"] Nov 01 14:43:30.509952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.509928 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-75g2x"] Nov 01 14:43:30.592689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.592641 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:43:30.592893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.592717 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-xrbx9" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:43:30.623911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.623881 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6"] Nov 01 14:43:30.650317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.650188 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-cksk6"] Nov 01 14:43:30.665605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.665578 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:43:30.665753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.665640 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:43:30.822245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.822192 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" Nov 01 14:43:30.822659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.822188 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5" event={"ID":"c4ab0917-810c-476c-a535-9a031d74c629","Type":"ContainerDied","Data":"536f801cc36e6bd0a47c7aac29479db0e836f1a8243d31ec736dc85493399930"} Nov 01 14:43:30.822659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:30.822362 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536f801cc36e6bd0a47c7aac29479db0e836f1a8243d31ec736dc85493399930" Nov 01 14:43:31.054748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.054717 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr"] Nov 01 14:43:31.058046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.058016 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4542b32b-slcnr"] Nov 01 14:43:31.067043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.067017 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Nov 01 14:43:31.067156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.067077 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Nov 01 14:43:31.083126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.083103 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" path="/var/lib/kubelet/pods/007c0d84-c42e-4d3b-ad3c-cef5e43bf883/volumes" Nov 01 14:43:31.083671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.083656 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" path="/var/lib/kubelet/pods/0ff736db-b409-4a27-b198-0108ac29adc5/volumes" Nov 01 14:43:31.084098 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.084070 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770" path="/var/lib/kubelet/pods/1ecfc18f-cb9a-4bcf-a3e7-6cb5a463b770/volumes" Nov 01 14:43:31.084849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.084828 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" path="/var/lib/kubelet/pods/37576851-40f6-42ab-bc50-be00a24bc5e1/volumes" Nov 01 14:43:31.085278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:31.085264 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" path="/var/lib/kubelet/pods/7ce6f30b-c719-4010-a12b-6ef7de59940c/volumes" Nov 01 14:43:31.114972 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:43:31.114932 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:43:32.070497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:32.070467 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:43:32.070908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:32.070885 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:43:32.071479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:32.071456 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:43:33.545372 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:33.545332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:33.719626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:33.719592 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:38.147445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:38.147411 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:38.697975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:38.697910 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:39.347262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.346630 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855"] Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347632 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347655 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347668 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347679 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347710 5473 cpu_manager.go:414] "containerMap: removing container" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347719 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347765 5473 cpu_manager.go:414] "containerMap: removing container" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347773 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347788 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c4ab0917-810c-476c-a535-9a031d74c629" containerName="revoke-keys" Nov 01 14:43:39.347817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347795 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ab0917-810c-476c-a535-9a031d74c629" containerName="revoke-keys" Nov 01 14:43:39.348539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347949 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce6f30b-c719-4010-a12b-6ef7de59940c" containerName="istio-proxy" Nov 01 14:43:39.348539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347963 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff736db-b409-4a27-b198-0108ac29adc5" containerName="istio-proxy" Nov 01 14:43:39.348539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347977 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="007c0d84-c42e-4d3b-ad3c-cef5e43bf883" containerName="istio-proxy" Nov 01 14:43:39.348539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.347996 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="37576851-40f6-42ab-bc50-be00a24bc5e1" containerName="istio-proxy" Nov 01 14:43:39.348539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.348005 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ab0917-810c-476c-a535-9a031d74c629" containerName="revoke-keys" Nov 01 14:43:39.398277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.380338 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.398713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.398582 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855"] Nov 01 14:43:39.441704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.441671 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-71e39a-openshift-default-dockercfg-hjbg7\"" Nov 01 14:43:39.444854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.443943 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:39.474930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.474899 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475000 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsncv\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475066 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475293 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475351 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475529 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.475609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.475587 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577008 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577061 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577100 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577161 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577185 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsncv\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577247 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577286 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577382 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577425 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577459 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577668 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577752 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.577812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.577791 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.578273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.578250 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.579418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.579395 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.579770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.579750 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.647579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.647500 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsncv\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.647738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.647615 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token\") pod \"e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.707424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.707388 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:39.795376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:39.792175 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:40.147293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:40.147265 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855"] Nov 01 14:43:40.176392 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:43:40.176362 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb442603a_e1bb_460f_aa23_7a0de15c2f84.slice/crio-da6e1871ba925f6e0b59815f60c3256929476a2950fcb0522973d4f48980c828 WatchSource:0}: Error finding container da6e1871ba925f6e0b59815f60c3256929476a2950fcb0522973d4f48980c828: Status 404 returned error can't find the container with id da6e1871ba925f6e0b59815f60c3256929476a2950fcb0522973d4f48980c828 Nov 01 14:43:40.178894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:40.178861 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:40.178977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:40.178927 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:40.178977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:40.178956 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:40.934987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:40.934950 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" event={"ID":"b442603a-e1bb-460f-aa23-7a0de15c2f84","Type":"ContainerStarted","Data":"99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad"} Nov 01 14:43:40.935396 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:40.935013 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" event={"ID":"b442603a-e1bb-460f-aa23-7a0de15c2f84","Type":"ContainerStarted","Data":"da6e1871ba925f6e0b59815f60c3256929476a2950fcb0522973d4f48980c828"} Nov 01 14:43:41.105387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:41.105342 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podStartSLOduration=2.105329865 podStartE2EDuration="2.105329865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:41.104682668 +0000 UTC m=+1560.162834517" watchObservedRunningTime="2026-11-01 14:43:41.105329865 +0000 UTC m=+1560.163481714" Nov 01 14:43:41.707933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:41.707901 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:41.712869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:41.712846 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:41.943263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:41.943232 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:41.944290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:41.944271 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:43:42.000523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.000457 5473 scope.go:122] "RemoveContainer" containerID="c3331878798f952d4dc51b8b18acedead13ac3e631d70f52ad036d9fb00337b6" Nov 01 14:43:42.010714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.010694 5473 scope.go:122] "RemoveContainer" containerID="0c813b8ec0821e98653076b25e1437c2feb7ec48a812fef4047fb096d624a58c" Nov 01 14:43:42.019785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.019766 5473 scope.go:122] "RemoveContainer" containerID="5c6643e54d69e81bbac4a28bd7271615b4a8e60c58562695ca470f40ab39f1dc" Nov 01 14:43:42.028623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.028607 5473 scope.go:122] "RemoveContainer" containerID="319e23f5ddefbe628222e16b54dab8364de80fd83f4eda0236d17ea89685c676" Nov 01 14:43:42.037102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.037084 5473 scope.go:122] "RemoveContainer" containerID="c3b3be8bd3a4a2bc34132b108265372777377b43550518f0086464a15cc654c1" Nov 01 14:43:42.045543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.045526 5473 scope.go:122] "RemoveContainer" containerID="1d0f0cee62eb32f7045ec78704de2bb292ec261fe17a3cc02e2fa5f2897b4f11" Nov 01 14:43:42.053827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.053808 5473 scope.go:122] "RemoveContainer" containerID="ccf0454909e74c8a75dd80574a4963c84eaa80b60bd52081965e2969b86293c0" Nov 01 14:43:42.064415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.064393 5473 scope.go:122] "RemoveContainer" containerID="b1140351160bac2f06cfacb95882759b151c4acf347a7da0db4bdeb2fcedba4b" Nov 01 14:43:42.076782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.076762 5473 scope.go:122] "RemoveContainer" containerID="8cf41cda46b60338735c5f7e1116569a79af0e643e9077f4cfdb5ffe0493466a" Nov 01 14:43:42.901407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:42.901379 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:43.870255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:43.870227 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:44.076439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:44.076407 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:44.438558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:44.438531 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:47.054197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.054159 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm"] Nov 01 14:43:47.061376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.061346 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:47.118270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.110031 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-079f4d72-h8zdm"] Nov 01 14:43:47.226167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.226132 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:47.408200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.408115 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js"] Nov 01 14:43:47.414943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.414920 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.467184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.467156 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js"] Nov 01 14:43:47.498156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.498127 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g275f\" (UniqueName: \"kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.498351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.498258 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.563377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.563344 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-71e39a-dockercfg-bzqfk\"" Nov 01 14:43:47.564228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.564187 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-71e39a\"" Nov 01 14:43:47.602891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.600051 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7"] Nov 01 14:43:47.602891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.600933 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.602891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.601164 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g275f\" (UniqueName: \"kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.602891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.602753 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.611879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.608155 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.639684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.639609 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7"] Nov 01 14:43:47.674302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.671188 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g275f\" (UniqueName: \"kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f\") pod \"payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.702356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.702313 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w94b\" (UniqueName: \"kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b\") pod \"payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.702530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.702416 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.729193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.728536 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:47.805124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.803861 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w94b\" (UniqueName: \"kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b\") pod \"payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.805124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.803974 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.805124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.804816 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.861620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.861596 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w94b\" (UniqueName: \"kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b\") pod \"payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:47.944107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:47.944035 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:48.071808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:48.071773 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:48.116809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:48.116776 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js"] Nov 01 14:43:48.313117 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:43:48.313088 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26622bc_b8da_4d22_8f65_a297597266da.slice/crio-d46d354250e0418134148242d019e9b9d75fda2aee9e396ea901fd105e97bd83 WatchSource:0}: Error finding container d46d354250e0418134148242d019e9b9d75fda2aee9e396ea901fd105e97bd83: Status 404 returned error can't find the container with id d46d354250e0418134148242d019e9b9d75fda2aee9e396ea901fd105e97bd83 Nov 01 14:43:48.317154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:48.317131 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7"] Nov 01 14:43:48.778989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:48.778959 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h"] Nov 01 14:43:48.779157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:48.779140 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" containerID="cri-o://c5c6fb20b55987e7afec9ab6f43710acf680de3a56ad6e98913df5471426e6eb" gracePeriod=30 Nov 01 14:43:49.021739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.021698 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" event={"ID":"c26622bc-b8da-4d22-8f65-a297597266da","Type":"ContainerStarted","Data":"1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211"} Nov 01 14:43:49.021952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.021895 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:43:49.021952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.021937 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" event={"ID":"c26622bc-b8da-4d22-8f65-a297597266da","Type":"ContainerStarted","Data":"d46d354250e0418134148242d019e9b9d75fda2aee9e396ea901fd105e97bd83"} Nov 01 14:43:49.023516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.023487 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" event={"ID":"3f7fce6f-bebb-4816-82e0-1397baf4c4d8","Type":"ContainerStarted","Data":"bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249"} Nov 01 14:43:49.023634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.023534 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" event={"ID":"3f7fce6f-bebb-4816-82e0-1397baf4c4d8","Type":"ContainerStarted","Data":"d71748d2854e2f2b28a6d66a146358eadb3226a8c829c1e38489fe5f7d33813b"} Nov 01 14:43:49.023634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.023562 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:43:49.082706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.082640 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b52187-cf28-43ba-b3b5-a11f7f24b598" path="/var/lib/kubelet/pods/61b52187-cf28-43ba-b3b5-a11f7f24b598/volumes" Nov 01 14:43:49.138752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.138704 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" podStartSLOduration=2.138692128 podStartE2EDuration="2.138692128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:49.136693235 +0000 UTC m=+1568.194845084" watchObservedRunningTime="2026-11-01 14:43:49.138692128 +0000 UTC m=+1568.196843997" Nov 01 14:43:49.256128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.256066 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" podStartSLOduration=2.256047993 podStartE2EDuration="2.256047993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:49.253090001 +0000 UTC m=+1568.311241851" watchObservedRunningTime="2026-11-01 14:43:49.256047993 +0000 UTC m=+1568.314199844" Nov 01 14:43:49.982206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.982176 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm"] Nov 01 14:43:49.987593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:49.987576 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033436 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033540 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033612 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033676 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033743 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033805 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033839 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.033898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.033886 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvsx\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.039921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.039898 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-26a0c1-openshift-default-dockercfg-xrpzs\"" Nov 01 14:43:50.094288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.094256 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n"] Nov 01 14:43:50.098939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.098911 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-500188c1-8v22n"] Nov 01 14:43:50.101057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.101027 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm"] Nov 01 14:43:50.135476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135444 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.135719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135520 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.135719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135613 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvsx\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.135719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135671 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.135719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135717 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.135961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135833 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.135961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.135902 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.136082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.136003 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.136082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.136039 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.136294 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.136268 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.136559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.136514 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.136798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.136774 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.137007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.136983 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.137266 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.137200 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.139664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.139640 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.140394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.140357 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.192543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.192517 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.192643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.192605 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvsx\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.301973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.301953 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:50.717832 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:43:50.717800 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b3e7a2_2b81_496d_a2dd_118fc29bb065.slice/crio-990e0a1519d727e1310ee410559dccdb0aba5d2766d91796a63e25d6eebd9f72 WatchSource:0}: Error finding container 990e0a1519d727e1310ee410559dccdb0aba5d2766d91796a63e25d6eebd9f72: Status 404 returned error can't find the container with id 990e0a1519d727e1310ee410559dccdb0aba5d2766d91796a63e25d6eebd9f72 Nov 01 14:43:50.720445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.720411 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:50.720527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.720480 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:50.720527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.720510 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:50.721894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:50.721875 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm"] Nov 01 14:43:51.039170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.039132 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" event={"ID":"b0b3e7a2-2b81-496d-a2dd-118fc29bb065","Type":"ContainerStarted","Data":"cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66"} Nov 01 14:43:51.039360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.039192 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" event={"ID":"b0b3e7a2-2b81-496d-a2dd-118fc29bb065","Type":"ContainerStarted","Data":"990e0a1519d727e1310ee410559dccdb0aba5d2766d91796a63e25d6eebd9f72"} Nov 01 14:43:51.082595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.082568 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231285ed-84db-4302-bc1c-fbd7761ff94c" path="/var/lib/kubelet/pods/231285ed-84db-4302-bc1c-fbd7761ff94c/volumes" Nov 01 14:43:51.158047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.158001 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podStartSLOduration=2.15798688 podStartE2EDuration="2.15798688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:43:51.155332989 +0000 UTC m=+1570.213484839" watchObservedRunningTime="2026-11-01 14:43:51.15798688 +0000 UTC m=+1570.216138729" Nov 01 14:43:51.188128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.188099 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:51.302781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.302716 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:51.304486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.304462 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 01 14:43:51.304605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.304506 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 01 14:43:51.348321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:51.348289 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:52.102300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:52.101524 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:52.302770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:52.302240 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 01 14:43:52.302770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:52.302309 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 01 14:43:53.302774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:53.302734 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 01 14:43:53.303162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:53.302798 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 01 14:43:53.974343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:53.974289 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:54.116832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.116809 5473 generic.go:356] "Generic (PLEG): container finished" podID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerID="c5c6fb20b55987e7afec9ab6f43710acf680de3a56ad6e98913df5471426e6eb" exitCode=0 Nov 01 14:43:54.116933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.116882 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" event={"ID":"5e80f9aa-1292-4bf7-9850-f3dd331e377d","Type":"ContainerDied","Data":"c5c6fb20b55987e7afec9ab6f43710acf680de3a56ad6e98913df5471426e6eb"} Nov 01 14:43:54.116988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.116937 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" event={"ID":"5e80f9aa-1292-4bf7-9850-f3dd331e377d","Type":"ContainerDied","Data":"f6a3d140e59fc8379fd93ec582dc19d53589278a4f38fee1f98eaba032214ccb"} Nov 01 14:43:54.116988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.116958 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a3d140e59fc8379fd93ec582dc19d53589278a4f38fee1f98eaba032214ccb" Nov 01 14:43:54.149277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.144391 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:43:54.192315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.192283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:54.211069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211036 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211120 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211159 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211179 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211207 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211258 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211406 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211510 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211630 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket\") pod \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\" (UID: \"5e80f9aa-1292-4bf7-9850-f3dd331e377d\") " Nov 01 14:43:54.211762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211509 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:54.211762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211710 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:54.212040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212014 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:54.212127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.211927 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:43:54.212127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212114 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:54.212480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212458 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.212570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212484 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.212570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212499 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.212570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212514 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.212570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.212529 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.213527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.213503 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:43:54.213714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.213691 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:54.213785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.213724 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "kube-api-access-rngh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:43:54.213785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.213767 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy" pod "5e80f9aa-1292-4bf7-9850-f3dd331e377d" (UID: "5e80f9aa-1292-4bf7-9850-f3dd331e377d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:43:54.302807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.302768 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 01 14:43:54.303299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.302828 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 01 14:43:54.314142 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.314092 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rngh4\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-kube-api-access-rngh4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.314468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.314405 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.314468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.314436 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.314468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.314451 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e80f9aa-1292-4bf7-9850-f3dd331e377d-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:43:54.477883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:54.477797 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:55.135606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.124212 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h" Nov 01 14:43:55.303187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.303154 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 01 14:43:55.303574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.303204 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 01 14:43:55.321719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.321692 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h"] Nov 01 14:43:55.372713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.372633 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-zx76h"] Nov 01 14:43:55.421319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.421282 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:55.599690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:55.599654 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:56.304239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:56.302543 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Nov 01 14:43:56.304239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:56.302614 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Nov 01 14:43:56.845198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:56.845164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:57.086742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:57.086703 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" path="/var/lib/kubelet/pods/5e80f9aa-1292-4bf7-9850-f3dd331e377d/volumes" Nov 01 14:43:57.307351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:57.307308 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:57.307805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:57.307740 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:57.308402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:57.308383 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:43:57.330303 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:57.330278 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:57.934788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:57.934750 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:58.222882 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.222809 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:58.687061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.687029 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj"] Nov 01 14:43:58.687679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.687664 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" Nov 01 14:43:58.687724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.687681 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" Nov 01 14:43:58.687808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.687799 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e80f9aa-1292-4bf7-9850-f3dd331e377d" containerName="istio-proxy" Nov 01 14:43:58.692756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.692733 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.770946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.770891 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:58.779395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.779359 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj"] Nov 01 14:43:58.811633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.811569 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-25ce0b-openshift-default-dockercfg-2nlps\"" Nov 01 14:43:58.823113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.823057 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.823615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.823593 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.824201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.824183 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.824955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.824916 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.825099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.825083 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.825274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.825255 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.825402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.825387 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwzc\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.825585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.825566 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.825835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.825802 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927257 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927318 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927447 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927572 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927547 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927615 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927661 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927717 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927723 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927744 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwzc\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.927822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.927813 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.928042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.928007 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.928189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.928130 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.928309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.928228 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.928376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.928353 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.929906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.929886 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.930252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.930231 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.945686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.945631 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:58.987464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.987435 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:58.987619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:58.987579 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwzc\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc\") pod \"e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:59.009577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:59.009546 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:43:59.216174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:59.216040 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:43:59.451476 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:43:59.451442 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc47b33b_ef0e_44d5_9097_d4fa413682f5.slice/crio-e60ec6503bdab91a97c087dccbfba6734680f04fb413c8cbde8ba4a7ecaaecdc WatchSource:0}: Error finding container e60ec6503bdab91a97c087dccbfba6734680f04fb413c8cbde8ba4a7ecaaecdc: Status 404 returned error can't find the container with id e60ec6503bdab91a97c087dccbfba6734680f04fb413c8cbde8ba4a7ecaaecdc Nov 01 14:43:59.454177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:59.454144 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:59.454319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:59.454296 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:59.454390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:59.454352 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:43:59.455570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:43:59.455547 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj"] Nov 01 14:44:00.033625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.033599 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:44:00.034029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.033673 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:44:00.064527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.064484 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:00.172088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.172046 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c"] Nov 01 14:44:00.191045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.190989 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.227373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.226737 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c"] Nov 01 14:44:00.228753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.228712 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" event={"ID":"bc47b33b-ef0e-44d5-9097-d4fa413682f5","Type":"ContainerStarted","Data":"ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0"} Nov 01 14:44:00.228878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.228775 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" event={"ID":"bc47b33b-ef0e-44d5-9097-d4fa413682f5","Type":"ContainerStarted","Data":"e60ec6503bdab91a97c087dccbfba6734680f04fb413c8cbde8ba4a7ecaaecdc"} Nov 01 14:44:00.229918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.229873 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:00.235669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.235632 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc"] Nov 01 14:44:00.235887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.235864 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" containerID="cri-o://fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef" gracePeriod=30 Nov 01 14:44:00.239894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.239875 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z"] Nov 01 14:44:00.240076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.240056 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" containerID="cri-o://a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f" gracePeriod=30 Nov 01 14:44:00.263452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.263403 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-cbac83-openshift-default-dockercfg-mc5zf\"" Nov 01 14:44:00.276196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.275577 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898"] Nov 01 14:44:00.276196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.275944 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" containerID="cri-o://93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d" gracePeriod=30 Nov 01 14:44:00.278227 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.278191 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9"] Nov 01 14:44:00.278511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.278482 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" containerID="cri-o://e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2" gracePeriod=30 Nov 01 14:44:00.279514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.279488 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.279757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.279737 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.279896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.279879 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.280060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.280043 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.281524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.281498 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.281702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.281685 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.281829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.281814 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.281966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.281948 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2c9m\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.282114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.282098 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384090 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384125 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2c9m\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384166 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384256 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384652 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384689 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384748 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384781 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.384838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384806 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.385104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.384937 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.385104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.385045 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.385346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.385327 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.385494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.385383 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.387299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.387202 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.421724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.421685 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.441461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.441432 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.441649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.441474 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2c9m\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m\") pod \"e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.538880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.538852 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:00.570956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.570908 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:00.919669 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:44:00.919634 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53af0abe_a1e5_49cc_919f_00324f85a09b.slice/crio-db19b9fd74df26fcfdf31d056ea0c7fc9b06db695aab467be2530c6cc39e54db WatchSource:0}: Error finding container db19b9fd74df26fcfdf31d056ea0c7fc9b06db695aab467be2530c6cc39e54db: Status 404 returned error can't find the container with id db19b9fd74df26fcfdf31d056ea0c7fc9b06db695aab467be2530c6cc39e54db Nov 01 14:44:00.922210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.922176 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:44:00.922349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.922265 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:44:00.922349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.922295 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:44:00.922602 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:00.922577 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c"] Nov 01 14:44:01.003506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.003463 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podStartSLOduration=3.003445982 podStartE2EDuration="3.003445982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:01.001932079 +0000 UTC m=+1580.060083931" watchObservedRunningTime="2026-11-01 14:44:01.003445982 +0000 UTC m=+1580.061597832" Nov 01 14:44:01.010119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.010096 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:44:01.016094 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.016074 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:44:01.114412 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:01.114376 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:44:01.241957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.241855 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" event={"ID":"53af0abe-a1e5-49cc-919f-00324f85a09b","Type":"ContainerStarted","Data":"77700b1e13952cda63766458fbedeb372d6c8c79741e41cd8795bb9bc68c576e"} Nov 01 14:44:01.241957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.241919 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" event={"ID":"53af0abe-a1e5-49cc-919f-00324f85a09b","Type":"ContainerStarted","Data":"db19b9fd74df26fcfdf31d056ea0c7fc9b06db695aab467be2530c6cc39e54db"} Nov 01 14:44:01.242273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.242248 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:44:01.243072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.243050 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:44:01.358648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.358597 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podStartSLOduration=1.358583524 podStartE2EDuration="1.358583524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:01.357889142 +0000 UTC m=+1580.416040993" watchObservedRunningTime="2026-11-01 14:44:01.358583524 +0000 UTC m=+1580.416735426" Nov 01 14:44:01.539185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:01.539122 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:02.177943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:02.177885 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:02.379418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:02.379386 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:02.545697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:02.545668 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:03.270241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:03.269572 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:03.271019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:03.270997 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:44:03.754633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:03.754537 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:03.971834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:03.971801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:04.055717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:04.055688 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc"] Nov 01 14:44:04.104703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:04.104676 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5801d564-n4crc"] Nov 01 14:44:04.143038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:04.142996 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:04.979670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:04.979639 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:44:04.980056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:04.979692 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:44:05.082013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.081984 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfa8616-7302-4634-9fb4-0809868ffff6" path="/var/lib/kubelet/pods/4dfa8616-7302-4634-9fb4-0809868ffff6/volumes" Nov 01 14:44:05.090339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.090309 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:05.265592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.265561 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:05.438637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.438599 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:05.580186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.580159 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:44:05.640962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.640941 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:44:05.644987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.644964 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:44:05.649591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.649574 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:44:05.673014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.672994 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673114 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673149 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673285 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673306 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.673353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673346 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673408 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673421 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.673469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673439 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673525 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673557 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy\") pod \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\" (UID: \"9bc2ec48-c811-4935-8a44-32edbcd9eeb6\") " Nov 01 14:44:05.673661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673622 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.673661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673639 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.673661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.673646 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:44:05.674054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.674032 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.674141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.674061 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.674141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.674077 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.674141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.674091 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.674141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.674105 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.675555 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.675528 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:44:05.675702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.675674 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.675702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.675687 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.675818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.675702 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff" pod "9bc2ec48-c811-4935-8a44-32edbcd9eeb6" (UID: "9bc2ec48-c811-4935-8a44-32edbcd9eeb6"). InnerVolumeSpecName "kube-api-access-n2qff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.775096 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775065 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.775307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775127 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.775307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775160 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.775307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775189 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775227 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775266 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775330 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775355 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775390 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775404 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775477 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775525 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775567 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775604 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775636 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775650 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775696 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775729 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775761 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775787 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775821 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775803 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775880 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775881 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775909 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:44:05.775974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.775975 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776016 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert\") pod \"7142a33e-409f-4cfe-b164-2b5a919255da\" (UID: \"7142a33e-409f-4cfe-b164-2b5a919255da\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776125 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776177 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776226 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776245 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo\") pod \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\" (UID: \"20d7eb3e-40d3-403a-b8dd-9c75c167848f\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776278 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776281 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776336 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo\") pod \"bbf1df67-e60a-4173-a438-2931bd477cd3\" (UID: \"bbf1df67-e60a-4173-a438-2931bd477cd3\") " Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776445 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.776925 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777406 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777430 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777445 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777458 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777474 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qff\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-kube-api-access-n2qff\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777489 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777502 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777516 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1df67-e60a-4173-a438-2931bd477cd3-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777531 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777546 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777561 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777576 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777592 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bc2ec48-c811-4935-8a44-32edbcd9eeb6-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777819 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.777998 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.778314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.778267 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.779494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.778569 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:44:05.779494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.778809 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.779601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.779512 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "kube-api-access-9vphp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.779601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.779537 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:44:05.779601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.779565 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.779871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.779842 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy" pod "bbf1df67-e60a-4173-a438-2931bd477cd3" (UID: "bbf1df67-e60a-4173-a438-2931bd477cd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.780180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.780153 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.780387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.780362 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.781542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.781504 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:44:05.781870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.781823 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "kube-api-access-59hcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.781951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.781912 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:44:05.782015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.781977 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.782085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.782061 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo" pod "7142a33e-409f-4cfe-b164-2b5a919255da" (UID: "7142a33e-409f-4cfe-b164-2b5a919255da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:44:05.783065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.783043 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.783718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.783699 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s" pod "20d7eb3e-40d3-403a-b8dd-9c75c167848f" (UID: "20d7eb3e-40d3-403a-b8dd-9c75c167848f"). InnerVolumeSpecName "kube-api-access-4gw7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:05.883047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883013 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vphp\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-kube-api-access-9vphp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883047 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7142a33e-409f-4cfe-b164-2b5a919255da-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883063 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883072 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883082 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883091 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59hcf\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-kube-api-access-59hcf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883102 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883115 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883129 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883141 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7142a33e-409f-4cfe-b164-2b5a919255da-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883159 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gw7s\" (UniqueName: \"kubernetes.io/projected/20d7eb3e-40d3-403a-b8dd-9c75c167848f-kube-api-access-4gw7s\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883168 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7142a33e-409f-4cfe-b164-2b5a919255da-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883206 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7142a33e-409f-4cfe-b164-2b5a919255da-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883235 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883248 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20d7eb3e-40d3-403a-b8dd-9c75c167848f-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883258 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20d7eb3e-40d3-403a-b8dd-9c75c167848f-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883270 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:05.883333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:05.883287 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1df67-e60a-4173-a438-2931bd477cd3-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:06.291979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.291933 5473 generic.go:356] "Generic (PLEG): container finished" podID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerID="93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d" exitCode=0 Nov 01 14:44:06.292436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.292001 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" Nov 01 14:44:06.292436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.292027 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" event={"ID":"9bc2ec48-c811-4935-8a44-32edbcd9eeb6","Type":"ContainerDied","Data":"93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d"} Nov 01 14:44:06.292436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.292091 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898" event={"ID":"9bc2ec48-c811-4935-8a44-32edbcd9eeb6","Type":"ContainerDied","Data":"ccb2c4333bc8714014606b75c8fb62f94f0be28f4fecb2b140ab87f24c02760c"} Nov 01 14:44:06.292436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.292126 5473 scope.go:122] "RemoveContainer" containerID="93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d" Nov 01 14:44:06.293741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.293719 5473 generic.go:356] "Generic (PLEG): container finished" podID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerID="fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef" exitCode=0 Nov 01 14:44:06.293860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.293749 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" event={"ID":"bbf1df67-e60a-4173-a438-2931bd477cd3","Type":"ContainerDied","Data":"fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef"} Nov 01 14:44:06.293860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.293775 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" Nov 01 14:44:06.293860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.293796 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc" event={"ID":"bbf1df67-e60a-4173-a438-2931bd477cd3","Type":"ContainerDied","Data":"b1fa10399a304d4c1cd38a96b40ba0b950d397703eee660f5430b57c6424d966"} Nov 01 14:44:06.295933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.295911 5473 generic.go:356] "Generic (PLEG): container finished" podID="7142a33e-409f-4cfe-b164-2b5a919255da" containerID="e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2" exitCode=0 Nov 01 14:44:06.296040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.295991 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" Nov 01 14:44:06.296040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.296001 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" event={"ID":"7142a33e-409f-4cfe-b164-2b5a919255da","Type":"ContainerDied","Data":"e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2"} Nov 01 14:44:06.296176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.296066 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9" event={"ID":"7142a33e-409f-4cfe-b164-2b5a919255da","Type":"ContainerDied","Data":"46a98e484e5d954948963431414efd52d36f43f275d84da242b6a125dac6c93f"} Nov 01 14:44:06.297544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.297522 5473 generic.go:356] "Generic (PLEG): container finished" podID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerID="a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f" exitCode=0 Nov 01 14:44:06.297655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.297579 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" event={"ID":"20d7eb3e-40d3-403a-b8dd-9c75c167848f","Type":"ContainerDied","Data":"a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f"} Nov 01 14:44:06.297655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.297614 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" event={"ID":"20d7eb3e-40d3-403a-b8dd-9c75c167848f","Type":"ContainerDied","Data":"1466ee603dfe2b7fc1c42294caeaed02575ea4f840fc95c02f180f367c6ae56c"} Nov 01 14:44:06.297753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.297687 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z" Nov 01 14:44:06.305941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.305923 5473 scope.go:122] "RemoveContainer" containerID="93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d" Nov 01 14:44:06.306260 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:06.306201 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d\": container with ID starting with 93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d not found: ID does not exist" containerID="93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d" Nov 01 14:44:06.306358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.306268 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d"} err="failed to get container status \"93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d\": rpc error: code = NotFound desc = could not find container \"93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d\": container with ID starting with 93ff5187ae1d8169bdc228e7bdd46e74525cc42909a229baa487d0c48407a83d not found: ID does not exist" Nov 01 14:44:06.306358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.306292 5473 scope.go:122] "RemoveContainer" containerID="fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef" Nov 01 14:44:06.317628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.317608 5473 scope.go:122] "RemoveContainer" containerID="fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef" Nov 01 14:44:06.317913 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:06.317892 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef\": container with ID starting with fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef not found: ID does not exist" containerID="fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef" Nov 01 14:44:06.317963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.317919 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef"} err="failed to get container status \"fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef\": rpc error: code = NotFound desc = could not find container \"fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef\": container with ID starting with fea60f050a8b8152a41637bbec63187cc0daa42dca41ef1b0abdad94790202ef not found: ID does not exist" Nov 01 14:44:06.317963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.317935 5473 scope.go:122] "RemoveContainer" containerID="e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2" Nov 01 14:44:06.326902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.326887 5473 scope.go:122] "RemoveContainer" containerID="6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5" Nov 01 14:44:06.335627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.335611 5473 scope.go:122] "RemoveContainer" containerID="e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2" Nov 01 14:44:06.335877 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:06.335858 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2\": container with ID starting with e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2 not found: ID does not exist" containerID="e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2" Nov 01 14:44:06.335919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.335884 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2"} err="failed to get container status \"e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2\": rpc error: code = NotFound desc = could not find container \"e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2\": container with ID starting with e6efd4f2436147658ad49750249c649a31e468b4bbd5a6bbac9730bd32a101d2 not found: ID does not exist" Nov 01 14:44:06.335919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.335899 5473 scope.go:122] "RemoveContainer" containerID="6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5" Nov 01 14:44:06.336156 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:06.336139 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5\": container with ID starting with 6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5 not found: ID does not exist" containerID="6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5" Nov 01 14:44:06.336193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.336162 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5"} err="failed to get container status \"6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5\": rpc error: code = NotFound desc = could not find container \"6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5\": container with ID starting with 6c8ebb8a0992726718fc6b8ef224645903ea23845f6feedd9b63e9943debdef5 not found: ID does not exist" Nov 01 14:44:06.336193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.336177 5473 scope.go:122] "RemoveContainer" containerID="a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f" Nov 01 14:44:06.345002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.344986 5473 scope.go:122] "RemoveContainer" containerID="a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f" Nov 01 14:44:06.345257 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:06.345233 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f\": container with ID starting with a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f not found: ID does not exist" containerID="a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f" Nov 01 14:44:06.345312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.345266 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f"} err="failed to get container status \"a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f\": rpc error: code = NotFound desc = could not find container \"a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f\": container with ID starting with a1f9c0dbb703efa31f30168ab5127b934a71f68910a2585f2fa62ee03ff6023f not found: ID does not exist" Nov 01 14:44:06.458107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.458074 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898"] Nov 01 14:44:06.461464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.461443 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-mr898"] Nov 01 14:44:06.570474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.570391 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc"] Nov 01 14:44:06.575979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.575955 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-8b6kc"] Nov 01 14:44:06.685783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.685756 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9"] Nov 01 14:44:06.688871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.688850 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-qckr9"] Nov 01 14:44:06.798949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.798916 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z"] Nov 01 14:44:06.802109 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:06.802087 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-ggm4z"] Nov 01 14:44:07.082618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:07.082576 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" path="/var/lib/kubelet/pods/20d7eb3e-40d3-403a-b8dd-9c75c167848f/volumes" Nov 01 14:44:07.083146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:07.083123 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" path="/var/lib/kubelet/pods/7142a33e-409f-4cfe-b164-2b5a919255da/volumes" Nov 01 14:44:07.083659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:07.083639 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" path="/var/lib/kubelet/pods/9bc2ec48-c811-4935-8a44-32edbcd9eeb6/volumes" Nov 01 14:44:07.084561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:07.084540 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" path="/var/lib/kubelet/pods/bbf1df67-e60a-4173-a438-2931bd477cd3/volumes" Nov 01 14:44:08.060660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.060629 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:08.219867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.219836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:08.442117 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442027 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp"] Nov 01 14:44:08.442732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442710 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442733 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442751 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442758 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442784 5473 cpu_manager.go:414] "containerMap: removing container" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442794 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442811 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" Nov 01 14:44:08.442836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442819 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442845 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.442852 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.443007 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf1df67-e60a-4173-a438-2931bd477cd3" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.443024 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.443036 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d7eb3e-40d3-403a-b8dd-9c75c167848f" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.443048 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc2ec48-c811-4935-8a44-32edbcd9eeb6" containerName="istio-proxy" Nov 01 14:44:08.443180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.443059 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7142a33e-409f-4cfe-b164-2b5a919255da" containerName="istio-proxy" Nov 01 14:44:08.448637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.448617 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.502768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.500043 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp"] Nov 01 14:44:08.549601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.549572 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-25ce0b\"" Nov 01 14:44:08.562742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.562687 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-25ce0b-dockercfg-62rjw\"" Nov 01 14:44:08.608540 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.607237 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r"] Nov 01 14:44:08.612731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.612707 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.622512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.622466 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.622679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.622577 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2z84\" (UniqueName: \"kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.659729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.659698 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r"] Nov 01 14:44:08.724310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.724210 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.724448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.724333 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.724448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.724434 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2z84\" (UniqueName: \"kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.724528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.724465 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnqd\" (UniqueName: \"kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd\") pod \"payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.725062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.725042 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.781534 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.781502 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2z84\" (UniqueName: \"kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84\") pod \"payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:08.825418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.825375 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.825587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.825510 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnqd\" (UniqueName: \"kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd\") pod \"payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.826041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.826021 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.882556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.882527 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnqd\" (UniqueName: \"kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd\") pod \"payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:08.938162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:08.938134 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:09.061975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:09.061939 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:09.306286 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:44:09.306259 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883ea872_216e_488f_a1a7_2eb1e6faf615.slice/crio-8b995ea7712fe43e4b8efb0920ff3308fba034e33baa548904619fdfc0b924f2 WatchSource:0}: Error finding container 8b995ea7712fe43e4b8efb0920ff3308fba034e33baa548904619fdfc0b924f2: Status 404 returned error can't find the container with id 8b995ea7712fe43e4b8efb0920ff3308fba034e33baa548904619fdfc0b924f2 Nov 01 14:44:09.311291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:09.311262 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r"] Nov 01 14:44:09.330370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:09.330340 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" event={"ID":"883ea872-216e-488f-a1a7-2eb1e6faf615","Type":"ContainerStarted","Data":"8b995ea7712fe43e4b8efb0920ff3308fba034e33baa548904619fdfc0b924f2"} Nov 01 14:44:09.435960 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:44:09.435925 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf152497f_f9d8_488a_9abd_4cdc59c79e0f.slice/crio-756a631cb9f84d07ab47cfb7ae9259a786644bca7f959768837b18281072652c WatchSource:0}: Error finding container 756a631cb9f84d07ab47cfb7ae9259a786644bca7f959768837b18281072652c: Status 404 returned error can't find the container with id 756a631cb9f84d07ab47cfb7ae9259a786644bca7f959768837b18281072652c Nov 01 14:44:09.438715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:09.438690 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp"] Nov 01 14:44:10.054732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.054573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:10.338714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.338624 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" event={"ID":"f152497f-f9d8-488a-9abd-4cdc59c79e0f","Type":"ContainerStarted","Data":"1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e"} Nov 01 14:44:10.338714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.338680 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" event={"ID":"f152497f-f9d8-488a-9abd-4cdc59c79e0f","Type":"ContainerStarted","Data":"756a631cb9f84d07ab47cfb7ae9259a786644bca7f959768837b18281072652c"} Nov 01 14:44:10.338714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.338708 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:10.340402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.340375 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" event={"ID":"883ea872-216e-488f-a1a7-2eb1e6faf615","Type":"ContainerStarted","Data":"72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df"} Nov 01 14:44:10.340510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.340454 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:10.457594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.457541 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" podStartSLOduration=2.457527618 podStartE2EDuration="2.457527618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:10.453670789 +0000 UTC m=+1589.511822639" watchObservedRunningTime="2026-11-01 14:44:10.457527618 +0000 UTC m=+1589.515679468" Nov 01 14:44:10.476682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.476651 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:10.571205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.571146 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" podStartSLOduration=2.571128854 podStartE2EDuration="2.571128854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:10.568397477 +0000 UTC m=+1589.626549328" watchObservedRunningTime="2026-11-01 14:44:10.571128854 +0000 UTC m=+1589.629280704" Nov 01 14:44:10.646069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.645988 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:10.851565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.851534 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p"] Nov 01 14:44:10.856186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.856163 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:10.862511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.862485 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:10.862679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.862661 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8f8v\" (UniqueName: \"kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:10.910437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.910343 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p"] Nov 01 14:44:10.930465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.911613 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cbac83\"" Nov 01 14:44:10.973240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.973132 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cbac83-dockercfg-pt2lg\"" Nov 01 14:44:10.993321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.993282 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8f8v\" (UniqueName: \"kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:10.993488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.993421 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:10.994586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:10.994556 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:11.020354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.020319 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh"] Nov 01 14:44:11.027101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.027075 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.050494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.050462 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8f8v\" (UniqueName: \"kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v\") pod \"payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:11.085173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.085146 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh"] Nov 01 14:44:11.094649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.094612 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2tb2\" (UniqueName: \"kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2\") pod \"payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.094793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.094760 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.176817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.176752 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:11.195619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.195583 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2tb2\" (UniqueName: \"kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2\") pod \"payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.195753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.195725 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.196597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.196573 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.252546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.252514 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2tb2\" (UniqueName: \"kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2\") pod \"payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.347210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.347179 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:11.546263 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:44:11.546230 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28f5824_7bc3_4fed_be98_8fe481d8f680.slice/crio-38091e0c29f5ccae9fd3f197171f352d6c02d0a67a1b8f316d94150af2229498 WatchSource:0}: Error finding container 38091e0c29f5ccae9fd3f197171f352d6c02d0a67a1b8f316d94150af2229498: Status 404 returned error can't find the container with id 38091e0c29f5ccae9fd3f197171f352d6c02d0a67a1b8f316d94150af2229498 Nov 01 14:44:11.550054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.550032 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p"] Nov 01 14:44:11.714445 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:44:11.714408 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8846e1_bd9d_49f5_842f_095f8e61b171.slice/crio-922a648d8906928b01d12e06bbe6439a3e371795bff32b4897179c69d98daab8 WatchSource:0}: Error finding container 922a648d8906928b01d12e06bbe6439a3e371795bff32b4897179c69d98daab8: Status 404 returned error can't find the container with id 922a648d8906928b01d12e06bbe6439a3e371795bff32b4897179c69d98daab8 Nov 01 14:44:11.718633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:11.718608 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh"] Nov 01 14:44:12.358736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.358695 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" event={"ID":"d28f5824-7bc3-4fed-be98-8fe481d8f680","Type":"ContainerStarted","Data":"4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483"} Nov 01 14:44:12.359253 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.358786 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" event={"ID":"d28f5824-7bc3-4fed-be98-8fe481d8f680","Type":"ContainerStarted","Data":"38091e0c29f5ccae9fd3f197171f352d6c02d0a67a1b8f316d94150af2229498"} Nov 01 14:44:12.359253 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.358817 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:12.360317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.360295 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" event={"ID":"dd8846e1-bd9d-49f5-842f-095f8e61b171","Type":"ContainerStarted","Data":"03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d"} Nov 01 14:44:12.360411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.360340 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" event={"ID":"dd8846e1-bd9d-49f5-842f-095f8e61b171","Type":"ContainerStarted","Data":"922a648d8906928b01d12e06bbe6439a3e371795bff32b4897179c69d98daab8"} Nov 01 14:44:12.360411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.360384 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:12.473018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.472958 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" podStartSLOduration=2.472943374 podStartE2EDuration="2.472943374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:12.471757869 +0000 UTC m=+1591.529909724" watchObservedRunningTime="2026-11-01 14:44:12.472943374 +0000 UTC m=+1591.531095225" Nov 01 14:44:12.592098 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:12.590573 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" podStartSLOduration=2.590551897 podStartE2EDuration="2.590551897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:12.585488135 +0000 UTC m=+1591.643639989" watchObservedRunningTime="2026-11-01 14:44:12.590551897 +0000 UTC m=+1591.648703749" Nov 01 14:44:14.796981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:14.796944 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:19.177933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:19.177897 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:20.222978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.222953 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9"] Nov 01 14:44:20.228905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.228885 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.317913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.317892 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.317924 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.317947 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.318061 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318187 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.318142 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.318208 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.318267 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mzt\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.318288 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.318463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.318323 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.335824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.335802 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9"] Nov 01 14:44:20.419996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.419962 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420005 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420024 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420065 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420118 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420179 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420420 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420458 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420523 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420679 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mzt\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420715 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420762 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.420968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.420889 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.421069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.421052 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.422786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.422767 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.422871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.422853 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.477240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.477146 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.477381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.477247 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mzt\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt\") pod \"e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.544587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.544552 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:20.962719 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:44:20.962688 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8e867d_76f2_46e8_b270_65b5436b9ad0.slice/crio-b45453d66ef1de0fc02325ae7195c9c5602235c600d505a1e6e81a6c17cdadfc WatchSource:0}: Error finding container b45453d66ef1de0fc02325ae7195c9c5602235c600d505a1e6e81a6c17cdadfc: Status 404 returned error can't find the container with id b45453d66ef1de0fc02325ae7195c9c5602235c600d505a1e6e81a6c17cdadfc Nov 01 14:44:20.965455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.965415 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:44:20.965536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.965501 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:44:20.965574 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.965543 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:44:20.966084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:20.966059 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9"] Nov 01 14:44:21.352689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:21.352662 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:44:21.353113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:21.352740 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:44:21.430391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:21.430360 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" event={"ID":"cd8e867d-76f2-46e8-b270-65b5436b9ad0","Type":"ContainerStarted","Data":"5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62"} Nov 01 14:44:21.430538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:21.430415 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" event={"ID":"cd8e867d-76f2-46e8-b270-65b5436b9ad0","Type":"ContainerStarted","Data":"b45453d66ef1de0fc02325ae7195c9c5602235c600d505a1e6e81a6c17cdadfc"} Nov 01 14:44:21.544641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:21.544612 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:21.698950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:21.698837 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podStartSLOduration=1.698818656 podStartE2EDuration="1.698818656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:44:21.695586246 +0000 UTC m=+1600.753738096" watchObservedRunningTime="2026-11-01 14:44:21.698818656 +0000 UTC m=+1600.756970505" Nov 01 14:44:22.547239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:22.546973 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 01 14:44:22.547239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:22.547059 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 01 14:44:23.370875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:23.370845 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:44:23.371092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:23.370927 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:44:23.550615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:23.550581 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 01 14:44:23.551118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:23.550640 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 01 14:44:23.797040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:23.797011 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:24.547789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:24.547754 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 01 14:44:24.548091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:24.548059 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 01 14:44:25.545556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:25.545518 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 01 14:44:25.546149 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:25.546117 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 01 14:44:25.958296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:25.958180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:26.550170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:26.549689 5473 patch_prober.go:29] interesting pod/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 01 14:44:26.550170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:26.549758 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 01 14:44:27.316351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:27.316318 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:27.549036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:27.548996 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:27.549498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:27.549484 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:27.550239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:27.550203 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:44:28.749048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:28.749013 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:31.108289 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:31.108256 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:44:32.675238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:32.675179 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5"] Nov 01 14:44:32.782052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:32.782020 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw"] Nov 01 14:44:33.048853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:33.048809 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:33.244687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:33.244620 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:33.405291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:33.405158 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:33.561152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:33.561121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:37.676281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.676237 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" podUID="4263e1a9-35d8-4a47-a735-041fb36ea5ea" containerName="payload-processing" containerID="cri-o://d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f" gracePeriod=25 Nov 01 14:44:37.782877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.782821 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" podUID="8ff3f97f-5781-4714-8dde-a811c8d87223" containerName="payload-pre-processing" containerID="cri-o://601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0" gracePeriod=25 Nov 01 14:44:37.961640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.961611 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:44:37.983923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.983893 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw\" (UniqueName: \"kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw\") pod \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " Nov 01 14:44:37.984085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.983983 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume\") pod \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\" (UID: \"4263e1a9-35d8-4a47-a735-041fb36ea5ea\") " Nov 01 14:44:37.984697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.984665 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume" pod "4263e1a9-35d8-4a47-a735-041fb36ea5ea" (UID: "4263e1a9-35d8-4a47-a735-041fb36ea5ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:44:37.984867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.984847 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4263e1a9-35d8-4a47-a735-041fb36ea5ea-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:37.988273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:37.988243 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw" pod "4263e1a9-35d8-4a47-a735-041fb36ea5ea" (UID: "4263e1a9-35d8-4a47-a735-041fb36ea5ea"). InnerVolumeSpecName "kube-api-access-nhddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:38.051587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.051564 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:44:38.085148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.085118 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:38.085489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.085462 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw\" (UniqueName: \"kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw\") pod \"8ff3f97f-5781-4714-8dde-a811c8d87223\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " Nov 01 14:44:38.085626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.085527 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume\") pod \"8ff3f97f-5781-4714-8dde-a811c8d87223\" (UID: \"8ff3f97f-5781-4714-8dde-a811c8d87223\") " Nov 01 14:44:38.085942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.085916 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume" pod "8ff3f97f-5781-4714-8dde-a811c8d87223" (UID: "8ff3f97f-5781-4714-8dde-a811c8d87223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:44:38.086139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.086118 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ff3f97f-5781-4714-8dde-a811c8d87223-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:38.086252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.086142 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhddw\" (UniqueName: \"kubernetes.io/projected/4263e1a9-35d8-4a47-a735-041fb36ea5ea-kube-api-access-nhddw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:38.088185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.088159 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw" pod "8ff3f97f-5781-4714-8dde-a811c8d87223" (UID: "8ff3f97f-5781-4714-8dde-a811c8d87223"). InnerVolumeSpecName "kube-api-access-jsftw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:44:38.187395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.187265 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsftw\" (UniqueName: \"kubernetes.io/projected/8ff3f97f-5781-4714-8dde-a811c8d87223-kube-api-access-jsftw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:44:38.586720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.586689 5473 generic.go:356] "Generic (PLEG): container finished" podID="8ff3f97f-5781-4714-8dde-a811c8d87223" containerID="601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0" exitCode=0 Nov 01 14:44:38.586955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.586726 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" Nov 01 14:44:38.586955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.586764 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" event={"ID":"8ff3f97f-5781-4714-8dde-a811c8d87223","Type":"ContainerDied","Data":"601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0"} Nov 01 14:44:38.586955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.586822 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw" event={"ID":"8ff3f97f-5781-4714-8dde-a811c8d87223","Type":"ContainerDied","Data":"3d999ac5d695165d51c4551a0ba753c06ceda060b16776e4e32b1a1e447aff5c"} Nov 01 14:44:38.586955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.586854 5473 scope.go:122] "RemoveContainer" containerID="601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0" Nov 01 14:44:38.588323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.588297 5473 generic.go:356] "Generic (PLEG): container finished" podID="4263e1a9-35d8-4a47-a735-041fb36ea5ea" containerID="d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f" exitCode=0 Nov 01 14:44:38.588445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.588346 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" Nov 01 14:44:38.588445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.588386 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" event={"ID":"4263e1a9-35d8-4a47-a735-041fb36ea5ea","Type":"ContainerDied","Data":"d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f"} Nov 01 14:44:38.588537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.588443 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5" event={"ID":"4263e1a9-35d8-4a47-a735-041fb36ea5ea","Type":"ContainerDied","Data":"0e11aadf9882aee6e941f7a8683780fbe0f8418a8cbe751345c0a484743e243c"} Nov 01 14:44:38.598564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.598542 5473 scope.go:122] "RemoveContainer" containerID="601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0" Nov 01 14:44:38.599262 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:38.599239 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0\": container with ID starting with 601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0 not found: ID does not exist" containerID="601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0" Nov 01 14:44:38.599374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.599268 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0"} err="failed to get container status \"601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0\": rpc error: code = NotFound desc = could not find container \"601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0\": container with ID starting with 601d7c3d5c94990fc220c5968aa7a3fa8a7c3d6f8a8b1bcaa3bcf8a24b5d61f0 not found: ID does not exist" Nov 01 14:44:38.599374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.599284 5473 scope.go:122] "RemoveContainer" containerID="d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f" Nov 01 14:44:38.608519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.608503 5473 scope.go:122] "RemoveContainer" containerID="d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f" Nov 01 14:44:38.608774 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:44:38.608756 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f\": container with ID starting with d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f not found: ID does not exist" containerID="d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f" Nov 01 14:44:38.608844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.608779 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f"} err="failed to get container status \"d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f\": rpc error: code = NotFound desc = could not find container \"d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f\": container with ID starting with d2d3db6c2ee02ef89d3f048f1fec50e1974457cb7e00b2c2875368129197df7f not found: ID does not exist" Nov 01 14:44:38.752513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.752483 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5"] Nov 01 14:44:38.755842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.755822 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0c4976-6776786597-l6sb5"] Nov 01 14:44:38.866170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.866095 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw"] Nov 01 14:44:38.869513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:38.869486 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0c4976-6f576bb44c-pp2pw"] Nov 01 14:44:39.082662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:39.082631 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4263e1a9-35d8-4a47-a735-041fb36ea5ea" path="/var/lib/kubelet/pods/4263e1a9-35d8-4a47-a735-041fb36ea5ea/volumes" Nov 01 14:44:39.083033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:39.083020 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff3f97f-5781-4714-8dde-a811c8d87223" path="/var/lib/kubelet/pods/8ff3f97f-5781-4714-8dde-a811c8d87223/volumes" Nov 01 14:44:39.669126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:39.669093 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:42.185866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.185836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:42.191565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.191545 5473 scope.go:122] "RemoveContainer" containerID="c5c6fb20b55987e7afec9ab6f43710acf680de3a56ad6e98913df5471426e6eb" Nov 01 14:44:42.201805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.201789 5473 scope.go:122] "RemoveContainer" containerID="4a260ccb64d018c4069753e59ba55a647285b7b8cfff5de51073db34d2704284" Nov 01 14:44:42.211189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.211166 5473 scope.go:122] "RemoveContainer" containerID="e831ceb6c3e3814dd8e14e6dea5204b79cda0106f109c0df11172070073cdd8c" Nov 01 14:44:42.222127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.222105 5473 scope.go:122] "RemoveContainer" containerID="f363bdd105e49559dc59ead089f271f1d9b1e61fb41e4b6877bdfbc06df44d84" Nov 01 14:44:42.230959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.230942 5473 scope.go:122] "RemoveContainer" containerID="643e28d582415e2ab41d49c7de07a680a018cfeea2bde7cff7a3879b88e59b9f" Nov 01 14:44:42.239482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.239466 5473 scope.go:122] "RemoveContainer" containerID="6908533dd9d56cb0017492b584e3dbc4b210d66fbed55b1929454577c825c4ba" Nov 01 14:44:42.336378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.336349 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:42.751741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:42.751700 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:46.022502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:46.022469 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:46.489718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:46.489635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:48.897344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:48.897287 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:51.507940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:51.507857 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:53.435643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:53.435610 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:57.940383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:57.940348 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:58.128930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:58.128887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:44:58.321493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:44:58.321461 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:00.165711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.165678 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4"] Nov 01 14:45:00.166328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.166312 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8ff3f97f-5781-4714-8dde-a811c8d87223" containerName="payload-pre-processing" Nov 01 14:45:00.166394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.166328 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff3f97f-5781-4714-8dde-a811c8d87223" containerName="payload-pre-processing" Nov 01 14:45:00.166394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.166346 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4263e1a9-35d8-4a47-a735-041fb36ea5ea" containerName="payload-processing" Nov 01 14:45:00.166394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.166351 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4263e1a9-35d8-4a47-a735-041fb36ea5ea" containerName="payload-processing" Nov 01 14:45:00.166494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.166460 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4263e1a9-35d8-4a47-a735-041fb36ea5ea" containerName="payload-processing" Nov 01 14:45:00.166494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.166483 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff3f97f-5781-4714-8dde-a811c8d87223" containerName="payload-pre-processing" Nov 01 14:45:00.173596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.173575 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp"] Nov 01 14:45:00.173707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.173646 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.177586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.177559 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w"] Nov 01 14:45:00.177708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.177672 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.181640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.181624 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.215852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.215833 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt"] Nov 01 14:45:00.219772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.219756 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2"] Nov 01 14:45:00.219893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.219877 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.224087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.224072 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2"] Nov 01 14:45:00.224242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.224228 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.228982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.228963 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl"] Nov 01 14:45:00.229335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.229130 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.234067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.234047 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4"] Nov 01 14:45:00.234160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.234090 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp"] Nov 01 14:45:00.234160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.234138 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.264686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.263939 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9c22\" (UniqueName: \"kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.264686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.264033 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:45:00.265821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.265782 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.265957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.265837 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2t42\" (UniqueName: \"kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42\") pod \"maas-api-key-cleanup-29892405-cfj4w\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.265957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.265876 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.265957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.265898 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jxk\" (UniqueName: \"kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk\") pod \"maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.265957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.265948 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t9cc\" (UniqueName: \"kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.266176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.265975 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.266176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266034 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.266176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv2w\" (UniqueName: \"kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w\") pod \"maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.266376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266240 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.266376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266283 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkq6\" (UniqueName: \"kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6\") pod \"maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.266376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266320 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4254l\" (UniqueName: \"kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l\") pod \"maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.266376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266359 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.266631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.266414 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892405-cfj4w\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.269307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.269284 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w"] Nov 01 14:45:00.273801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.273780 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2"] Nov 01 14:45:00.277622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.277599 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt"] Nov 01 14:45:00.311192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.310888 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl"] Nov 01 14:45:00.319558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.319531 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2"] Nov 01 14:45:00.367657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367621 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.367845 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367680 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2t42\" (UniqueName: \"kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42\") pod \"maas-api-key-cleanup-29892405-cfj4w\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.367845 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367732 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.367845 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367769 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jxk\" (UniqueName: \"kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk\") pod \"maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.367845 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367819 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t9cc\" (UniqueName: \"kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.368100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367847 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.368100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.367905 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.368100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368014 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv2w\" (UniqueName: \"kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w\") pod \"maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.368385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368107 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.368385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368147 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkq6\" (UniqueName: \"kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6\") pod \"maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.368385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368182 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4254l\" (UniqueName: \"kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l\") pod \"maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.368385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368234 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.368385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368286 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892405-cfj4w\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.368734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368421 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9c22\" (UniqueName: \"kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.368734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.368586 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.369172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.369148 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.369726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.369675 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.369726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.369716 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.369948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.369850 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.370186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.370169 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.370464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.370438 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892405-cfj4w\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.427346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.427199 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9c22\" (UniqueName: \"kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.427548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.427520 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jxk\" (UniqueName: \"kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk\") pod \"maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.427865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.427830 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t9cc\" (UniqueName: \"kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.427977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.427911 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2t42\" (UniqueName: \"kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42\") pod \"maas-api-key-cleanup-29892405-cfj4w\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.428118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.428094 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4254l\" (UniqueName: \"kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l\") pod \"maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.428198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.428097 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkq6\" (UniqueName: \"kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6\") pod \"maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.428488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.428470 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv2w\" (UniqueName: \"kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w\") pod \"maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.490643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.490607 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:00.498058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.498030 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:00.505233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.504126 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:00.535637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.535611 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:00.545176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.544908 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:00.567072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.567000 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:00.587652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.584365 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:00.928416 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:45:00.928387 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode198cc6e_50c2_4994_9ce6_1c40f3d32b18.slice/crio-80f7f4879827a699c787f454e8a10656c1c496e1305b33707bb660d4106e6908 WatchSource:0}: Error finding container 80f7f4879827a699c787f454e8a10656c1c496e1305b33707bb660d4106e6908: Status 404 returned error can't find the container with id 80f7f4879827a699c787f454e8a10656c1c496e1305b33707bb660d4106e6908 Nov 01 14:45:00.933758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.933728 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4"] Nov 01 14:45:00.972747 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:45:00.972722 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ed7b64_b4d7_4f09_af34_ac1b2737a6d0.slice/crio-911cd3ee71162e50d279cd84f87761b31225fd28f92cef639f8ba235fcace60d WatchSource:0}: Error finding container 911cd3ee71162e50d279cd84f87761b31225fd28f92cef639f8ba235fcace60d: Status 404 returned error can't find the container with id 911cd3ee71162e50d279cd84f87761b31225fd28f92cef639f8ba235fcace60d Nov 01 14:45:00.978116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:00.978094 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp"] Nov 01 14:45:01.038954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.038929 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2"] Nov 01 14:45:01.045680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.045654 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl"] Nov 01 14:45:01.140038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.140010 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w"] Nov 01 14:45:01.150309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.150270 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2"] Nov 01 14:45:01.174784 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:45:01.174735 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7093fb3_a8c6_4f31_a246_495885c4450e.slice/crio-011c2153606610c25d6bc325f29fa3e95d6aa12e5afd5a2406f70d0b9965367d WatchSource:0}: Error finding container 011c2153606610c25d6bc325f29fa3e95d6aa12e5afd5a2406f70d0b9965367d: Status 404 returned error can't find the container with id 011c2153606610c25d6bc325f29fa3e95d6aa12e5afd5a2406f70d0b9965367d Nov 01 14:45:01.177775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.177753 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt"] Nov 01 14:45:01.183348 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:01.183285 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:45:01.835617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.835526 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerStarted","Data":"c1f16153e1aba66e960f00c5391a32d8c70c532c1d360de3cda2f2ccdc4f3dfc"} Nov 01 14:45:01.835617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.835578 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerStarted","Data":"f84705b8310eeb258af9c83b64ffc81f0b310463641e667c175457fe96bc1b52"} Nov 01 14:45:01.837275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.837250 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerStarted","Data":"886be5a0b53a8bab9afe1077934b57a696da0ec87158816f22dfe06b3bd951a5"} Nov 01 14:45:01.837423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.837292 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerStarted","Data":"911cd3ee71162e50d279cd84f87761b31225fd28f92cef639f8ba235fcace60d"} Nov 01 14:45:01.838867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.838847 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerStarted","Data":"84bd6aea3b9893ef473f3893e15948a420257086b38d1efbe37d4f88d7cb80bf"} Nov 01 14:45:01.838971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.838885 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerStarted","Data":"58d00e5659c3b9f991bce83c7d58abc6c9543f099e2468f5e8bc34041baf0f64"} Nov 01 14:45:01.840472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.840449 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerStarted","Data":"b338eab24a2291924131e20c1a2672cf6e23eaaec33f950cc92afbd3b8c9d6fe"} Nov 01 14:45:01.840610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.840489 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerStarted","Data":"011c2153606610c25d6bc325f29fa3e95d6aa12e5afd5a2406f70d0b9965367d"} Nov 01 14:45:01.841994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.841973 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerStarted","Data":"05dfdb7980a2611589643038a0611cb84378b6b27439091ed40951d6dbf30b9e"} Nov 01 14:45:01.842119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.842005 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerStarted","Data":"6d51543b77698d852f290642e7e39997a51516fe36908304c6b77e688ed3257c"} Nov 01 14:45:01.843495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.843475 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerStarted","Data":"0338978985d8ccbda91dfdc46ec63648b961f049972effa4bb4c186d67c4099c"} Nov 01 14:45:01.843582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.843511 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerStarted","Data":"80f7f4879827a699c787f454e8a10656c1c496e1305b33707bb660d4106e6908"} Nov 01 14:45:01.844970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.844950 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerStarted","Data":"da56119b2446c58478fd72d432006aea0af8ee61db8fb21f69097d4de233ab66"} Nov 01 14:45:01.845078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.844983 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerStarted","Data":"57288ec657965424602fe22737305846b810a79b36a91ec634b04520e358d899"} Nov 01 14:45:01.949335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:01.949280 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" podStartSLOduration=1.949261834 podStartE2EDuration="1.949261834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:01.948337682 +0000 UTC m=+1641.006489532" watchObservedRunningTime="2026-11-01 14:45:01.949261834 +0000 UTC m=+1641.007413688" Nov 01 14:45:02.069199 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.069142 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" podStartSLOduration=2.069127253 podStartE2EDuration="2.069127253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.068607508 +0000 UTC m=+1641.126759357" watchObservedRunningTime="2026-11-01 14:45:02.069127253 +0000 UTC m=+1641.127279103" Nov 01 14:45:02.184274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.184208 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" podStartSLOduration=2.184196273 podStartE2EDuration="2.184196273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.180731934 +0000 UTC m=+1641.238883781" watchObservedRunningTime="2026-11-01 14:45:02.184196273 +0000 UTC m=+1641.242348176" Nov 01 14:45:02.293910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.293864 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" podStartSLOduration=2.293853029 podStartE2EDuration="2.293853029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.293247128 +0000 UTC m=+1641.351398968" watchObservedRunningTime="2026-11-01 14:45:02.293853029 +0000 UTC m=+1641.352004921" Nov 01 14:45:02.406916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.406836 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" podStartSLOduration=2.4068233980000002 podStartE2EDuration="2.406823398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.405410486 +0000 UTC m=+1641.463562335" watchObservedRunningTime="2026-11-01 14:45:02.406823398 +0000 UTC m=+1641.464975247" Nov 01 14:45:02.519541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.519490 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" podStartSLOduration=2.519475336 podStartE2EDuration="2.519475336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.518836212 +0000 UTC m=+1641.576988061" watchObservedRunningTime="2026-11-01 14:45:02.519475336 +0000 UTC m=+1641.577627186" Nov 01 14:45:02.632234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.632174 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" podStartSLOduration=2.632161659 podStartE2EDuration="2.632161659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:45:02.631894361 +0000 UTC m=+1641.690046210" watchObservedRunningTime="2026-11-01 14:45:02.632161659 +0000 UTC m=+1641.690313513" Nov 01 14:45:02.867984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.867916 5473 generic.go:356] "Generic (PLEG): container finished" podID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerID="05dfdb7980a2611589643038a0611cb84378b6b27439091ed40951d6dbf30b9e" exitCode=60 Nov 01 14:45:02.868159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.868074 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerDied","Data":"05dfdb7980a2611589643038a0611cb84378b6b27439091ed40951d6dbf30b9e"} Nov 01 14:45:02.868610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.868582 5473 scope.go:122] "RemoveContainer" containerID="05dfdb7980a2611589643038a0611cb84378b6b27439091ed40951d6dbf30b9e" Nov 01 14:45:02.873758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.873503 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:02.876133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.876104 5473 generic.go:356] "Generic (PLEG): container finished" podID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerID="0338978985d8ccbda91dfdc46ec63648b961f049972effa4bb4c186d67c4099c" exitCode=60 Nov 01 14:45:02.876274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.876252 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerDied","Data":"0338978985d8ccbda91dfdc46ec63648b961f049972effa4bb4c186d67c4099c"} Nov 01 14:45:02.876715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.876689 5473 scope.go:122] "RemoveContainer" containerID="0338978985d8ccbda91dfdc46ec63648b961f049972effa4bb4c186d67c4099c" Nov 01 14:45:02.879755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.879732 5473 generic.go:356] "Generic (PLEG): container finished" podID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerID="da56119b2446c58478fd72d432006aea0af8ee61db8fb21f69097d4de233ab66" exitCode=60 Nov 01 14:45:02.879897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.879871 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerDied","Data":"da56119b2446c58478fd72d432006aea0af8ee61db8fb21f69097d4de233ab66"} Nov 01 14:45:02.880127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.880108 5473 scope.go:122] "RemoveContainer" containerID="da56119b2446c58478fd72d432006aea0af8ee61db8fb21f69097d4de233ab66" Nov 01 14:45:02.881665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.881642 5473 generic.go:356] "Generic (PLEG): container finished" podID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerID="c1f16153e1aba66e960f00c5391a32d8c70c532c1d360de3cda2f2ccdc4f3dfc" exitCode=60 Nov 01 14:45:02.881751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.881721 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerDied","Data":"c1f16153e1aba66e960f00c5391a32d8c70c532c1d360de3cda2f2ccdc4f3dfc"} Nov 01 14:45:02.881928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.881910 5473 scope.go:122] "RemoveContainer" containerID="c1f16153e1aba66e960f00c5391a32d8c70c532c1d360de3cda2f2ccdc4f3dfc" Nov 01 14:45:02.890105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.884784 5473 generic.go:356] "Generic (PLEG): container finished" podID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerID="886be5a0b53a8bab9afe1077934b57a696da0ec87158816f22dfe06b3bd951a5" exitCode=60 Nov 01 14:45:02.890105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.884907 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerDied","Data":"886be5a0b53a8bab9afe1077934b57a696da0ec87158816f22dfe06b3bd951a5"} Nov 01 14:45:02.890105 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.885077 5473 scope.go:122] "RemoveContainer" containerID="886be5a0b53a8bab9afe1077934b57a696da0ec87158816f22dfe06b3bd951a5" Nov 01 14:45:02.890725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.890692 5473 generic.go:356] "Generic (PLEG): container finished" podID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerID="84bd6aea3b9893ef473f3893e15948a420257086b38d1efbe37d4f88d7cb80bf" exitCode=7 Nov 01 14:45:02.891039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.891019 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerDied","Data":"84bd6aea3b9893ef473f3893e15948a420257086b38d1efbe37d4f88d7cb80bf"} Nov 01 14:45:02.891504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.891484 5473 scope.go:122] "RemoveContainer" containerID="84bd6aea3b9893ef473f3893e15948a420257086b38d1efbe37d4f88d7cb80bf" Nov 01 14:45:02.893874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.893806 5473 generic.go:356] "Generic (PLEG): container finished" podID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerID="b338eab24a2291924131e20c1a2672cf6e23eaaec33f950cc92afbd3b8c9d6fe" exitCode=60 Nov 01 14:45:02.893874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.893845 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerDied","Data":"b338eab24a2291924131e20c1a2672cf6e23eaaec33f950cc92afbd3b8c9d6fe"} Nov 01 14:45:02.894098 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:02.894083 5473 scope.go:122] "RemoveContainer" containerID="b338eab24a2291924131e20c1a2672cf6e23eaaec33f950cc92afbd3b8c9d6fe" Nov 01 14:45:03.063696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.063665 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:03.224000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.223950 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:03.903796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.903759 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerStarted","Data":"5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc"} Nov 01 14:45:03.905663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.905635 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerStarted","Data":"3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a"} Nov 01 14:45:03.907642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.907619 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerStarted","Data":"ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1"} Nov 01 14:45:03.909523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.909500 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerStarted","Data":"dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee"} Nov 01 14:45:03.911169 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.911142 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerStarted","Data":"f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0"} Nov 01 14:45:03.912970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.912945 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerStarted","Data":"a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4"} Nov 01 14:45:03.914575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:03.914558 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerStarted","Data":"157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e"} Nov 01 14:45:04.408980 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.408946 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode198cc6e_50c2_4994_9ce6_1c40f3d32b18.slice/crio-dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee.scope\": RecentStats: unable to find data in memory cache]" Nov 01 14:45:04.924178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.924138 5473 generic.go:356] "Generic (PLEG): container finished" podID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerID="157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e" exitCode=60 Nov 01 14:45:04.924397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.924202 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerDied","Data":"157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e"} Nov 01 14:45:04.924397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.924329 5473 scope.go:122] "RemoveContainer" containerID="886be5a0b53a8bab9afe1077934b57a696da0ec87158816f22dfe06b3bd951a5" Nov 01 14:45:04.927993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.924695 5473 scope.go:122] "RemoveContainer" containerID="157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e" Nov 01 14:45:04.927993 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.925091 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp_odh-ai-gateway-infra(83ed7b64-b4d7-4f09-af34-ac1b2737a6d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" Nov 01 14:45:04.928554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.928393 5473 generic.go:356] "Generic (PLEG): container finished" podID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerID="5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc" exitCode=7 Nov 01 14:45:04.928554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.928488 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerDied","Data":"5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc"} Nov 01 14:45:04.932236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.928880 5473 scope.go:122] "RemoveContainer" containerID="5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc" Nov 01 14:45:04.932236 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.929246 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2_odh-ai-gateway-infra(058a1791-1bce-4b5b-9248-a6a24cb36993)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" Nov 01 14:45:04.932236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.931504 5473 generic.go:356] "Generic (PLEG): container finished" podID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerID="3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a" exitCode=60 Nov 01 14:45:04.932236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.931580 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerDied","Data":"3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a"} Nov 01 14:45:04.932236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.931921 5473 scope.go:122] "RemoveContainer" containerID="3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a" Nov 01 14:45:04.932567 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.932250 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt_odh-ai-gateway-infra(d7093fb3-a8c6-4f31-a246-495885c4450e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" Nov 01 14:45:04.936233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.934580 5473 generic.go:356] "Generic (PLEG): container finished" podID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerID="ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1" exitCode=60 Nov 01 14:45:04.936233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.934683 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerDied","Data":"ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1"} Nov 01 14:45:04.936233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.934996 5473 scope.go:122] "RemoveContainer" containerID="ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1" Nov 01 14:45:04.936233 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.935260 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892405-cfj4w_odh-ai-gateway-infra(ac7906b7-1585-49a0-90f4-ed0adc65b92d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" Nov 01 14:45:04.940241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.937545 5473 generic.go:356] "Generic (PLEG): container finished" podID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerID="dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee" exitCode=60 Nov 01 14:45:04.940241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.937710 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerDied","Data":"dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee"} Nov 01 14:45:04.940241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.938016 5473 scope.go:122] "RemoveContainer" containerID="dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee" Nov 01 14:45:04.940241 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.938266 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4_odh-ai-gateway-infra(e198cc6e-50c2-4994-9ce6-1c40f3d32b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.940571 5473 generic.go:356] "Generic (PLEG): container finished" podID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerID="f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0" exitCode=60 Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.940717 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerDied","Data":"f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0"} Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.941106 5473 scope.go:122] "RemoveContainer" containerID="f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0" Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.941367 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl_odh-ai-gateway-infra(2f712cd5-7799-4d23-8f09-67fb58da9a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.943776 5473 generic.go:356] "Generic (PLEG): container finished" podID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerID="a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4" exitCode=60 Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.943845 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerDied","Data":"a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4"} Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.944304 5473 scope.go:122] "RemoveContainer" containerID="a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4" Nov 01 14:45:04.945246 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:04.944537 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2_odh-ai-gateway-infra(3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" Nov 01 14:45:04.959701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.959681 5473 scope.go:122] "RemoveContainer" containerID="84bd6aea3b9893ef473f3893e15948a420257086b38d1efbe37d4f88d7cb80bf" Nov 01 14:45:04.973002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.972978 5473 scope.go:122] "RemoveContainer" containerID="b338eab24a2291924131e20c1a2672cf6e23eaaec33f950cc92afbd3b8c9d6fe" Nov 01 14:45:04.986059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.986042 5473 scope.go:122] "RemoveContainer" containerID="05dfdb7980a2611589643038a0611cb84378b6b27439091ed40951d6dbf30b9e" Nov 01 14:45:04.998711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:04.998693 5473 scope.go:122] "RemoveContainer" containerID="0338978985d8ccbda91dfdc46ec63648b961f049972effa4bb4c186d67c4099c" Nov 01 14:45:05.011604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.011584 5473 scope.go:122] "RemoveContainer" containerID="da56119b2446c58478fd72d432006aea0af8ee61db8fb21f69097d4de233ab66" Nov 01 14:45:05.024206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.024186 5473 scope.go:122] "RemoveContainer" containerID="c1f16153e1aba66e960f00c5391a32d8c70c532c1d360de3cda2f2ccdc4f3dfc" Nov 01 14:45:05.959697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.959667 5473 scope.go:122] "RemoveContainer" containerID="f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0" Nov 01 14:45:05.960182 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.960026 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl_odh-ai-gateway-infra(2f712cd5-7799-4d23-8f09-67fb58da9a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" Nov 01 14:45:05.961547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.961526 5473 scope.go:122] "RemoveContainer" containerID="a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4" Nov 01 14:45:05.961870 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.961846 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2_odh-ai-gateway-infra(3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" Nov 01 14:45:05.963181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.963161 5473 scope.go:122] "RemoveContainer" containerID="157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e" Nov 01 14:45:05.963497 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.963474 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp_odh-ai-gateway-infra(83ed7b64-b4d7-4f09-af34-ac1b2737a6d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" Nov 01 14:45:05.964824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.964803 5473 scope.go:122] "RemoveContainer" containerID="5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc" Nov 01 14:45:05.965099 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.965081 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2_odh-ai-gateway-infra(058a1791-1bce-4b5b-9248-a6a24cb36993)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" Nov 01 14:45:05.966420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.966398 5473 scope.go:122] "RemoveContainer" containerID="3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a" Nov 01 14:45:05.966688 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.966665 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt_odh-ai-gateway-infra(d7093fb3-a8c6-4f31-a246-495885c4450e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" Nov 01 14:45:05.967834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.967816 5473 scope.go:122] "RemoveContainer" containerID="ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1" Nov 01 14:45:05.968089 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.968071 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892405-cfj4w_odh-ai-gateway-infra(ac7906b7-1585-49a0-90f4-ed0adc65b92d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" Nov 01 14:45:05.969279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:05.969262 5473 scope.go:122] "RemoveContainer" containerID="dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee" Nov 01 14:45:05.969501 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:05.969485 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4_odh-ai-gateway-infra(e198cc6e-50c2-4994-9ce6-1c40f3d32b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" Nov 01 14:45:07.021898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:07.021866 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:07.892325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:07.892289 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:08.728557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:08.728464 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:11.945144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:11.945040 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 14:45:11.945144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:11.945099 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 14:45:12.019595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.019564 5473 generic.go:356] "Generic (PLEG): container finished" podID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerID="99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad" exitCode=0 Nov 01 14:45:12.019741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.019636 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" event={"ID":"b442603a-e1bb-460f-aa23-7a0de15c2f84","Type":"ContainerDied","Data":"99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad"} Nov 01 14:45:12.020047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.020030 5473 scope.go:122] "RemoveContainer" containerID="99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad" Nov 01 14:45:12.021497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.021472 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:12.021559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.021540 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:12.021593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.021572 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:12.129785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.129752 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:12.299743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:12.299704 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:13.037250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:13.036346 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" event={"ID":"b442603a-e1bb-460f-aa23-7a0de15c2f84","Type":"ContainerStarted","Data":"1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830"} Nov 01 14:45:13.322675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:13.322592 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:13.554903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:13.554870 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:13.707614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:13.707529 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:45:13.707765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:13.707725 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 14:45:13.707822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:13.707772 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 14:45:14.080718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:14.080687 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:14.707856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:14.707819 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 14:45:14.708079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:14.707887 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 14:45:14.802503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:14.802460 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:15.074628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:15.074599 5473 scope.go:122] "RemoveContainer" containerID="a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4" Nov 01 14:45:15.707856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:15.707775 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 14:45:15.707856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:15.707827 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 14:45:16.075020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.074990 5473 scope.go:122] "RemoveContainer" containerID="5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc" Nov 01 14:45:16.075369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.075350 5473 scope.go:122] "RemoveContainer" containerID="157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e" Nov 01 14:45:16.078878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.078850 5473 generic.go:356] "Generic (PLEG): container finished" podID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerID="e2efe403c02ba7b2e690c9fb646952d9e33339046795ce23e3e482d07dd6fa46" exitCode=60 Nov 01 14:45:16.079004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.078885 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerDied","Data":"e2efe403c02ba7b2e690c9fb646952d9e33339046795ce23e3e482d07dd6fa46"} Nov 01 14:45:16.079004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.078945 5473 scope.go:122] "RemoveContainer" containerID="a3466e726a14611a25d988bcb549a7172549c7a5174b7f99cb578062ff2438f4" Nov 01 14:45:16.079343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.079206 5473 scope.go:122] "RemoveContainer" containerID="e2efe403c02ba7b2e690c9fb646952d9e33339046795ce23e3e482d07dd6fa46" Nov 01 14:45:16.079487 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:16.079454 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2_odh-ai-gateway-infra(3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" Nov 01 14:45:16.708114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.708033 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 14:45:16.708114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:16.708081 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 14:45:17.089454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.089422 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerStarted","Data":"637171056d583a3d155cd41f1f5366842c6a40b002712cba58585e075ae177dc"} Nov 01 14:45:17.091193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.091171 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerStarted","Data":"71c3809ce05008789d0e31f5af4e13ec2a4e5cff534d0962904ff9df0a4328a0"} Nov 01 14:45:17.264421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.264385 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2"] Nov 01 14:45:17.480607 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.480567 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:17.558970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.558938 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca\") pod \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " Nov 01 14:45:17.559180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.559161 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w\" (UniqueName: \"kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w\") pod \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\" (UID: \"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2\") " Nov 01 14:45:17.559358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.559330 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca" pod "3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" (UID: "3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:17.559724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.559700 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:17.561211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.561176 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w" pod "3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" (UID: "3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2"). InnerVolumeSpecName "kube-api-access-6rv2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:17.661321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.661278 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv2w\" (UniqueName: \"kubernetes.io/projected/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2-kube-api-access-6rv2w\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:17.708424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.708345 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Nov 01 14:45:17.708424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:17.708410 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Nov 01 14:45:18.074414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.074381 5473 scope.go:122] "RemoveContainer" containerID="dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee" Nov 01 14:45:18.074581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.074455 5473 scope.go:122] "RemoveContainer" containerID="3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a" Nov 01 14:45:18.074626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.074581 5473 scope.go:122] "RemoveContainer" containerID="ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1" Nov 01 14:45:18.074701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.074685 5473 scope.go:122] "RemoveContainer" containerID="f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0" Nov 01 14:45:18.113203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.097318 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:18.167339 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:18.167310 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7093fb3_a8c6_4f31_a246_495885c4450e.slice/crio-conmon-4a0423503cc2a6e010a9fcb1c13e32622e1c2eb77a32a730b55bda208ce45d54.scope\": RecentStats: unable to find data in memory cache]" Nov 01 14:45:18.170352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.170315 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" event={"ID":"3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2","Type":"ContainerDied","Data":"f84705b8310eeb258af9c83b64ffc81f0b310463641e667c175457fe96bc1b52"} Nov 01 14:45:18.175104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.170327 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2" Nov 01 14:45:18.175104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.172650 5473 generic.go:356] "Generic (PLEG): container finished" podID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerID="637171056d583a3d155cd41f1f5366842c6a40b002712cba58585e075ae177dc" exitCode=60 Nov 01 14:45:18.175104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.172785 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerDied","Data":"637171056d583a3d155cd41f1f5366842c6a40b002712cba58585e075ae177dc"} Nov 01 14:45:18.175104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.173153 5473 scope.go:122] "RemoveContainer" containerID="637171056d583a3d155cd41f1f5366842c6a40b002712cba58585e075ae177dc" Nov 01 14:45:18.175104 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:18.173438 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp_odh-ai-gateway-infra(83ed7b64-b4d7-4f09-af34-ac1b2737a6d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" Nov 01 14:45:18.175586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.175283 5473 scope.go:122] "RemoveContainer" containerID="e2efe403c02ba7b2e690c9fb646952d9e33339046795ce23e3e482d07dd6fa46" Nov 01 14:45:18.175783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.175749 5473 generic.go:356] "Generic (PLEG): container finished" podID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerID="71c3809ce05008789d0e31f5af4e13ec2a4e5cff534d0962904ff9df0a4328a0" exitCode=60 Nov 01 14:45:18.175861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.175824 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerDied","Data":"71c3809ce05008789d0e31f5af4e13ec2a4e5cff534d0962904ff9df0a4328a0"} Nov 01 14:45:18.176484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.176209 5473 scope.go:122] "RemoveContainer" containerID="71c3809ce05008789d0e31f5af4e13ec2a4e5cff534d0962904ff9df0a4328a0" Nov 01 14:45:18.176583 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:18.176509 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2_odh-ai-gateway-infra(058a1791-1bce-4b5b-9248-a6a24cb36993)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" Nov 01 14:45:18.194919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.194539 5473 scope.go:122] "RemoveContainer" containerID="157e6a8301bc42122f08b68d0ad811c61d88e87645b99bf2f11dd94e4b6cb96e" Nov 01 14:45:18.208590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.208572 5473 scope.go:122] "RemoveContainer" containerID="5e55d41d1dd42f7137b202a5e0680813f7d8375ebd0f097e85c86419506948fc" Nov 01 14:45:18.221498 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.221477 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp"] Nov 01 14:45:18.335843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.335757 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2"] Nov 01 14:45:18.456394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.456367 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2"] Nov 01 14:45:18.461449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.461429 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-18a879-29892405-d9hp2"] Nov 01 14:45:18.711813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.711789 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:45:18.712326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.712308 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:45:18.712946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:18.712926 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:45:19.098057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.098018 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" path="/var/lib/kubelet/pods/3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2/volumes" Nov 01 14:45:19.191594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.191566 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerStarted","Data":"4a0423503cc2a6e010a9fcb1c13e32622e1c2eb77a32a730b55bda208ce45d54"} Nov 01 14:45:19.193673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.193650 5473 generic.go:356] "Generic (PLEG): container finished" podID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerID="daf3edaff473db710cda1208b6178099252c115c32606fa56a149e459af4f6fa" exitCode=60 Nov 01 14:45:19.193794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.193723 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerDied","Data":"daf3edaff473db710cda1208b6178099252c115c32606fa56a149e459af4f6fa"} Nov 01 14:45:19.193794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.193783 5473 scope.go:122] "RemoveContainer" containerID="ab287d3ea43cca84d219b927938a6e636d87b6526ebe9a9e200c3b6ab484c6b1" Nov 01 14:45:19.194178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.194162 5473 scope.go:122] "RemoveContainer" containerID="daf3edaff473db710cda1208b6178099252c115c32606fa56a149e459af4f6fa" Nov 01 14:45:19.194556 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:19.194535 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892405-cfj4w_odh-ai-gateway-infra(ac7906b7-1585-49a0-90f4-ed0adc65b92d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" Nov 01 14:45:19.198703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.196959 5473 generic.go:356] "Generic (PLEG): container finished" podID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerID="c76fdda26ceaab41b8bfd86183150df1ca21950fcd479ed54299874137e55e8e" exitCode=60 Nov 01 14:45:19.198703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.197083 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerDied","Data":"c76fdda26ceaab41b8bfd86183150df1ca21950fcd479ed54299874137e55e8e"} Nov 01 14:45:19.198703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.197517 5473 scope.go:122] "RemoveContainer" containerID="c76fdda26ceaab41b8bfd86183150df1ca21950fcd479ed54299874137e55e8e" Nov 01 14:45:19.198703 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:19.197830 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4_odh-ai-gateway-infra(e198cc6e-50c2-4994-9ce6-1c40f3d32b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" Nov 01 14:45:19.203287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.200334 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerStarted","Data":"82fde717faad42442b6d8b7b3919cf4ed79efbbb330a9253f72fc34316b92a4a"} Nov 01 14:45:19.433743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.433719 5473 scope.go:122] "RemoveContainer" containerID="dd99bbf3c8b842648d734638d3b7c0bd424c0690a72bbc39752c2c9813ff01ee" Nov 01 14:45:19.439536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.439514 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:19.441589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.441568 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:19.628391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.628305 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l\" (UniqueName: \"kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l\") pod \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " Nov 01 14:45:19.628559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.628398 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca\") pod \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\" (UID: \"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0\") " Nov 01 14:45:19.628559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.628439 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca\") pod \"058a1791-1bce-4b5b-9248-a6a24cb36993\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " Nov 01 14:45:19.628679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.628573 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6\" (UniqueName: \"kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6\") pod \"058a1791-1bce-4b5b-9248-a6a24cb36993\" (UID: \"058a1791-1bce-4b5b-9248-a6a24cb36993\") " Nov 01 14:45:19.628789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.628761 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca" pod "83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" (UID: "83ed7b64-b4d7-4f09-af34-ac1b2737a6d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:19.628847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.628771 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca" pod "058a1791-1bce-4b5b-9248-a6a24cb36993" (UID: "058a1791-1bce-4b5b-9248-a6a24cb36993"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:19.629043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.629023 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:19.629110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.629051 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058a1791-1bce-4b5b-9248-a6a24cb36993-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:19.630557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.630530 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l" pod "83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" (UID: "83ed7b64-b4d7-4f09-af34-ac1b2737a6d0"). InnerVolumeSpecName "kube-api-access-4254l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:19.630661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.630585 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6" pod "058a1791-1bce-4b5b-9248-a6a24cb36993" (UID: "058a1791-1bce-4b5b-9248-a6a24cb36993"). InnerVolumeSpecName "kube-api-access-2hkq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:19.768238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.741056 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4254l\" (UniqueName: \"kubernetes.io/projected/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0-kube-api-access-4254l\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:19.769341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:19.769308 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hkq6\" (UniqueName: \"kubernetes.io/projected/058a1791-1bce-4b5b-9248-a6a24cb36993-kube-api-access-2hkq6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:20.212355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.212326 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" Nov 01 14:45:20.212548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.212355 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp" event={"ID":"83ed7b64-b4d7-4f09-af34-ac1b2737a6d0","Type":"ContainerDied","Data":"911cd3ee71162e50d279cd84f87761b31225fd28f92cef639f8ba235fcace60d"} Nov 01 14:45:20.212548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.212424 5473 scope.go:122] "RemoveContainer" containerID="637171056d583a3d155cd41f1f5366842c6a40b002712cba58585e075ae177dc" Nov 01 14:45:20.213963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.213942 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" event={"ID":"058a1791-1bce-4b5b-9248-a6a24cb36993","Type":"ContainerDied","Data":"58d00e5659c3b9f991bce83c7d58abc6c9543f099e2468f5e8bc34041baf0f64"} Nov 01 14:45:20.213963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.213961 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2" Nov 01 14:45:20.216009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.215986 5473 generic.go:356] "Generic (PLEG): container finished" podID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerID="4a0423503cc2a6e010a9fcb1c13e32622e1c2eb77a32a730b55bda208ce45d54" exitCode=60 Nov 01 14:45:20.216115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.216034 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerDied","Data":"4a0423503cc2a6e010a9fcb1c13e32622e1c2eb77a32a730b55bda208ce45d54"} Nov 01 14:45:20.216472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.216452 5473 scope.go:122] "RemoveContainer" containerID="4a0423503cc2a6e010a9fcb1c13e32622e1c2eb77a32a730b55bda208ce45d54" Nov 01 14:45:20.216747 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:20.216719 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt_odh-ai-gateway-infra(d7093fb3-a8c6-4f31-a246-495885c4450e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" Nov 01 14:45:20.221375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.221353 5473 generic.go:356] "Generic (PLEG): container finished" podID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerID="82fde717faad42442b6d8b7b3919cf4ed79efbbb330a9253f72fc34316b92a4a" exitCode=60 Nov 01 14:45:20.221571 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.221552 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerDied","Data":"82fde717faad42442b6d8b7b3919cf4ed79efbbb330a9253f72fc34316b92a4a"} Nov 01 14:45:20.222020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.222002 5473 scope.go:122] "RemoveContainer" containerID="82fde717faad42442b6d8b7b3919cf4ed79efbbb330a9253f72fc34316b92a4a" Nov 01 14:45:20.222344 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:20.222316 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl_odh-ai-gateway-infra(2f712cd5-7799-4d23-8f09-67fb58da9a70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" Nov 01 14:45:20.234960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.234937 5473 scope.go:122] "RemoveContainer" containerID="71c3809ce05008789d0e31f5af4e13ec2a4e5cff534d0962904ff9df0a4328a0" Nov 01 14:45:20.262128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.262107 5473 scope.go:122] "RemoveContainer" containerID="3133ab4bd10feeec6288767911ca8999ca5fd2fc785c6aa2518edbf9dbf47c0a" Nov 01 14:45:20.275598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.275577 5473 scope.go:122] "RemoveContainer" containerID="f958b95cfef935881ff403fc1876f968c2f9d81931257c69af99afbfcd2eabd0" Nov 01 14:45:20.476670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.475857 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:20.495239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.495195 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt"] Nov 01 14:45:20.516752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.516721 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2"] Nov 01 14:45:20.523126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.523098 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-25ce0b-29892405-crcn2"] Nov 01 14:45:20.683682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.683580 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl"] Nov 01 14:45:20.699012 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.698941 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:20.782935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.782901 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp"] Nov 01 14:45:20.786511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.786487 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-c9b0827c-29892405-5hnmp"] Nov 01 14:45:20.789515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.789497 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w"] Nov 01 14:45:20.920975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:20.920946 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4"] Nov 01 14:45:21.097936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.097897 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" path="/var/lib/kubelet/pods/058a1791-1bce-4b5b-9248-a6a24cb36993/volumes" Nov 01 14:45:21.098435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.098416 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" path="/var/lib/kubelet/pods/83ed7b64-b4d7-4f09-af34-ac1b2737a6d0/volumes" Nov 01 14:45:21.098799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.098781 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:21.207664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.207627 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca\") pod \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " Nov 01 14:45:21.207815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.207749 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42\" (UniqueName: \"kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42\") pod \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\" (UID: \"ac7906b7-1585-49a0-90f4-ed0adc65b92d\") " Nov 01 14:45:21.208751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.208725 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca" pod "ac7906b7-1585-49a0-90f4-ed0adc65b92d" (UID: "ac7906b7-1585-49a0-90f4-ed0adc65b92d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:21.211657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.211628 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42" pod "ac7906b7-1585-49a0-90f4-ed0adc65b92d" (UID: "ac7906b7-1585-49a0-90f4-ed0adc65b92d"). InnerVolumeSpecName "kube-api-access-p2t42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:21.301442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.301413 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:21.301791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.301770 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" Nov 01 14:45:21.301888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.301869 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w" event={"ID":"ac7906b7-1585-49a0-90f4-ed0adc65b92d","Type":"ContainerDied","Data":"6d51543b77698d852f290642e7e39997a51516fe36908304c6b77e688ed3257c"} Nov 01 14:45:21.301947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.301937 5473 scope.go:122] "RemoveContainer" containerID="daf3edaff473db710cda1208b6178099252c115c32606fa56a149e459af4f6fa" Nov 01 14:45:21.304143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.304120 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" event={"ID":"e198cc6e-50c2-4994-9ce6-1c40f3d32b18","Type":"ContainerDied","Data":"80f7f4879827a699c787f454e8a10656c1c496e1305b33707bb660d4106e6908"} Nov 01 14:45:21.304247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.304201 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4" Nov 01 14:45:21.310178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.310154 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7906b7-1585-49a0-90f4-ed0adc65b92d-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:21.310279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.310182 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2t42\" (UniqueName: \"kubernetes.io/projected/ac7906b7-1585-49a0-90f4-ed0adc65b92d-kube-api-access-p2t42\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:21.318264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.318239 5473 scope.go:122] "RemoveContainer" containerID="c76fdda26ceaab41b8bfd86183150df1ca21950fcd479ed54299874137e55e8e" Nov 01 14:45:21.381241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.377038 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:21.411347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.411267 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca\") pod \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " Nov 01 14:45:21.411529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.411409 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc\" (UniqueName: \"kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc\") pod \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\" (UID: \"e198cc6e-50c2-4994-9ce6-1c40f3d32b18\") " Nov 01 14:45:21.422713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.422671 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca" pod "e198cc6e-50c2-4994-9ce6-1c40f3d32b18" (UID: "e198cc6e-50c2-4994-9ce6-1c40f3d32b18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:21.470859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.466072 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc" pod "e198cc6e-50c2-4994-9ce6-1c40f3d32b18" (UID: "e198cc6e-50c2-4994-9ce6-1c40f3d32b18"). InnerVolumeSpecName "kube-api-access-2t9cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:21.522240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.513993 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:21.522240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.514034 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t9cc\" (UniqueName: \"kubernetes.io/projected/e198cc6e-50c2-4994-9ce6-1c40f3d32b18-kube-api-access-2t9cc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:21.900862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.899940 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w"] Nov 01 14:45:21.900862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.899985 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892405-cfj4w"] Nov 01 14:45:21.915401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.915374 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:21.991748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.991662 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22\" (UniqueName: \"kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22\") pod \"d7093fb3-a8c6-4f31-a246-495885c4450e\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " Nov 01 14:45:21.992010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.991986 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca\") pod \"d7093fb3-a8c6-4f31-a246-495885c4450e\" (UID: \"d7093fb3-a8c6-4f31-a246-495885c4450e\") " Nov 01 14:45:21.992102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.992081 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4"] Nov 01 14:45:21.993292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.993267 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca" pod "d7093fb3-a8c6-4f31-a246-495885c4450e" (UID: "d7093fb3-a8c6-4f31-a246-495885c4450e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:21.994863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.994840 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22" pod "d7093fb3-a8c6-4f31-a246-495885c4450e" (UID: "d7093fb3-a8c6-4f31-a246-495885c4450e"). InnerVolumeSpecName "kube-api-access-p9c22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:21.995229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:21.995199 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892405-vn5z4"] Nov 01 14:45:22.004143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.004122 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:22.093801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.093771 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk\" (UniqueName: \"kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk\") pod \"2f712cd5-7799-4d23-8f09-67fb58da9a70\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " Nov 01 14:45:22.093956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.093944 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca\") pod \"2f712cd5-7799-4d23-8f09-67fb58da9a70\" (UID: \"2f712cd5-7799-4d23-8f09-67fb58da9a70\") " Nov 01 14:45:22.094232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.094202 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7093fb3-a8c6-4f31-a246-495885c4450e-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:22.094360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.094252 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9c22\" (UniqueName: \"kubernetes.io/projected/d7093fb3-a8c6-4f31-a246-495885c4450e-kube-api-access-p9c22\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:22.094543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.094520 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca" pod "2f712cd5-7799-4d23-8f09-67fb58da9a70" (UID: "2f712cd5-7799-4d23-8f09-67fb58da9a70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:22.096014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.095991 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk" pod "2f712cd5-7799-4d23-8f09-67fb58da9a70" (UID: "2f712cd5-7799-4d23-8f09-67fb58da9a70"). InnerVolumeSpecName "kube-api-access-m9jxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:22.195314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.195281 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f712cd5-7799-4d23-8f09-67fb58da9a70-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:22.195314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.195313 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9jxk\" (UniqueName: \"kubernetes.io/projected/2f712cd5-7799-4d23-8f09-67fb58da9a70-kube-api-access-m9jxk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:22.319715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.319689 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" Nov 01 14:45:22.320427 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.319686 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt" event={"ID":"d7093fb3-a8c6-4f31-a246-495885c4450e","Type":"ContainerDied","Data":"011c2153606610c25d6bc325f29fa3e95d6aa12e5afd5a2406f70d0b9965367d"} Nov 01 14:45:22.320641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.320626 5473 scope.go:122] "RemoveContainer" containerID="4a0423503cc2a6e010a9fcb1c13e32622e1c2eb77a32a730b55bda208ce45d54" Nov 01 14:45:22.382166 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.381900 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" Nov 01 14:45:22.382385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.382362 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl" event={"ID":"2f712cd5-7799-4d23-8f09-67fb58da9a70","Type":"ContainerDied","Data":"57288ec657965424602fe22737305846b810a79b36a91ec634b04520e358d899"} Nov 01 14:45:22.383811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.383792 5473 scope.go:122] "RemoveContainer" containerID="82fde717faad42442b6d8b7b3919cf4ed79efbbb330a9253f72fc34316b92a4a" Nov 01 14:45:22.508634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.508600 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt"] Nov 01 14:45:22.513116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.512908 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892405-prkxt"] Nov 01 14:45:22.690350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.690261 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl"] Nov 01 14:45:22.694026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.694005 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-71e39a-29892405-mk6tl"] Nov 01 14:45:22.732241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:22.729856 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:23.084993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:23.084951 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" path="/var/lib/kubelet/pods/2f712cd5-7799-4d23-8f09-67fb58da9a70/volumes" Nov 01 14:45:23.085915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:23.085894 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" path="/var/lib/kubelet/pods/ac7906b7-1585-49a0-90f4-ed0adc65b92d/volumes" Nov 01 14:45:23.086407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:23.086389 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" path="/var/lib/kubelet/pods/d7093fb3-a8c6-4f31-a246-495885c4450e/volumes" Nov 01 14:45:23.087352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:23.087336 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" path="/var/lib/kubelet/pods/e198cc6e-50c2-4994-9ce6-1c40f3d32b18/volumes" Nov 01 14:45:26.981952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:26.981882 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:27.488560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:27.488527 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:29.453767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:29.453701 5473 generic.go:356] "Generic (PLEG): container finished" podID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerID="77700b1e13952cda63766458fbedeb372d6c8c79741e41cd8795bb9bc68c576e" exitCode=0 Nov 01 14:45:29.453767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:29.453763 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" event={"ID":"53af0abe-a1e5-49cc-919f-00324f85a09b","Type":"ContainerDied","Data":"77700b1e13952cda63766458fbedeb372d6c8c79741e41cd8795bb9bc68c576e"} Nov 01 14:45:29.454963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:29.454110 5473 scope.go:122] "RemoveContainer" containerID="77700b1e13952cda63766458fbedeb372d6c8c79741e41cd8795bb9bc68c576e" Nov 01 14:45:29.455662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:29.455630 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:29.455743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:29.455714 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:29.455809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:29.455765 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:30.462879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.462846 5473 generic.go:356] "Generic (PLEG): container finished" podID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerID="ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0" exitCode=0 Nov 01 14:45:30.463328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.462918 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" event={"ID":"bc47b33b-ef0e-44d5-9097-d4fa413682f5","Type":"ContainerDied","Data":"ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0"} Nov 01 14:45:30.463400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.463345 5473 scope.go:122] "RemoveContainer" containerID="ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0" Nov 01 14:45:30.464887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.464854 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:30.465001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.464939 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:30.465001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.464982 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:45:30.465543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.465464 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" event={"ID":"53af0abe-a1e5-49cc-919f-00324f85a09b","Type":"ContainerStarted","Data":"f7a440f0933c560e31ecfc9955c16e649c0566024c24323635195209a664eda2"} Nov 01 14:45:30.540961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.540880 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:45:30.544257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.542322 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:45:30.544257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:30.542373 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:45:31.109943 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:31.109908 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:45:31.247362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:31.247323 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:45:31.487307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:31.486344 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" event={"ID":"bc47b33b-ef0e-44d5-9097-d4fa413682f5","Type":"ContainerStarted","Data":"73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec"} Nov 01 14:45:31.548239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:31.547337 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:45:31.548239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:31.547402 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:45:32.009732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:32.009697 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:45:32.010854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:32.010819 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 14:45:32.010909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:32.010878 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 14:45:32.266099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:32.264414 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:32.540715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:32.540622 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:45:32.541330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:32.541298 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:45:33.010968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:33.010881 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 14:45:33.010968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:33.010942 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 14:45:33.273355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:33.273313 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:45:33.548720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:33.548627 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:45:33.549296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:33.549264 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:45:34.010675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:34.010570 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 14:45:34.010675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:34.010649 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 14:45:34.558240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:34.557613 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:45:34.558240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:34.557674 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:45:35.010827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:35.010731 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 14:45:35.010827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:35.010795 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 14:45:35.550779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:35.550743 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:45:35.551062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:35.551033 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:45:36.017235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:36.014659 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 01 14:45:36.017235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:36.014765 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 01 14:45:36.544591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:36.544557 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:45:36.545598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:36.545576 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:45:36.731172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:36.731132 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:37.015142 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:37.015111 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:45:37.015564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:37.015539 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:45:37.016150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:37.016133 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:45:41.177348 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:41.177316 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7"] Nov 01 14:45:41.283530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:41.283499 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js"] Nov 01 14:45:41.487519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:41.487435 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:41.673934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:41.673895 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:41.850117 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:41.850082 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:42.320864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:42.320831 5473 scope.go:122] "RemoveContainer" containerID="a42be2ea8d111e605fccab92b560ef4dd7404fa00b98f26859f31724b789229a" Nov 01 14:45:42.330767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:42.330745 5473 scope.go:122] "RemoveContainer" containerID="ebbd997afc6cb37c73e0870662099bf2e3680fe8bf9f47cb5fe8db3643c6a8c1" Nov 01 14:45:42.339854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:42.339835 5473 scope.go:122] "RemoveContainer" containerID="8fe81f0a2afd3bb4249cd0d951ebdd91165b8330957540bc6cc547fc0f215ce2" Nov 01 14:45:44.472252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:44.472202 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:45.278056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:45.278023 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:45.989324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:45.989294 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62"] Nov 01 14:45:46.095104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.095070 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq"] Nov 01 14:45:46.178417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.178354 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" podUID="c26622bc-b8da-4d22-8f65-a297597266da" containerName="payload-processing" containerID="cri-o://1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211" gracePeriod=25 Nov 01 14:45:46.286352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.286302 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" podUID="3f7fce6f-bebb-4816-82e0-1397baf4c4d8" containerName="payload-pre-processing" containerID="cri-o://bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249" gracePeriod=25 Nov 01 14:45:46.395718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.395640 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:46.472889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.472867 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:45:46.558962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.558856 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:46.611850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.611824 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b\" (UniqueName: \"kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b\") pod \"c26622bc-b8da-4d22-8f65-a297597266da\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " Nov 01 14:45:46.612005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.611894 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume\") pod \"c26622bc-b8da-4d22-8f65-a297597266da\" (UID: \"c26622bc-b8da-4d22-8f65-a297597266da\") " Nov 01 14:45:46.612379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.612345 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume" pod "c26622bc-b8da-4d22-8f65-a297597266da" (UID: "c26622bc-b8da-4d22-8f65-a297597266da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:46.612560 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.612536 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c26622bc-b8da-4d22-8f65-a297597266da-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:46.614163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.614141 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b" pod "c26622bc-b8da-4d22-8f65-a297597266da" (UID: "c26622bc-b8da-4d22-8f65-a297597266da"). InnerVolumeSpecName "kube-api-access-4w94b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:46.620276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.620256 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:45:46.629955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.629933 5473 generic.go:356] "Generic (PLEG): container finished" podID="3f7fce6f-bebb-4816-82e0-1397baf4c4d8" containerID="bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249" exitCode=0 Nov 01 14:45:46.630059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.629977 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" Nov 01 14:45:46.630059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.630014 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" event={"ID":"3f7fce6f-bebb-4816-82e0-1397baf4c4d8","Type":"ContainerDied","Data":"bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249"} Nov 01 14:45:46.630195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.630068 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js" event={"ID":"3f7fce6f-bebb-4816-82e0-1397baf4c4d8","Type":"ContainerDied","Data":"d71748d2854e2f2b28a6d66a146358eadb3226a8c829c1e38489fe5f7d33813b"} Nov 01 14:45:46.630195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.630096 5473 scope.go:122] "RemoveContainer" containerID="bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249" Nov 01 14:45:46.631777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.631756 5473 generic.go:356] "Generic (PLEG): container finished" podID="c26622bc-b8da-4d22-8f65-a297597266da" containerID="1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211" exitCode=0 Nov 01 14:45:46.631888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.631808 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" event={"ID":"c26622bc-b8da-4d22-8f65-a297597266da","Type":"ContainerDied","Data":"1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211"} Nov 01 14:45:46.631888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.631816 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" Nov 01 14:45:46.631888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.631848 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7" event={"ID":"c26622bc-b8da-4d22-8f65-a297597266da","Type":"ContainerDied","Data":"d46d354250e0418134148242d019e9b9d75fda2aee9e396ea901fd105e97bd83"} Nov 01 14:45:46.641451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.641366 5473 scope.go:122] "RemoveContainer" containerID="bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249" Nov 01 14:45:46.641729 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:46.641706 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249\": container with ID starting with bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249 not found: ID does not exist" containerID="bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249" Nov 01 14:45:46.641834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.641735 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249"} err="failed to get container status \"bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249\": rpc error: code = NotFound desc = could not find container \"bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249\": container with ID starting with bd2a0412220f20641744600f6dc776bcd6201b2c1acbf6bebcd0bf189c9c3249 not found: ID does not exist" Nov 01 14:45:46.641834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.641751 5473 scope.go:122] "RemoveContainer" containerID="1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211" Nov 01 14:45:46.651610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.651591 5473 scope.go:122] "RemoveContainer" containerID="1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211" Nov 01 14:45:46.651876 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:46.651860 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211\": container with ID starting with 1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211 not found: ID does not exist" containerID="1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211" Nov 01 14:45:46.651938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.651884 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211"} err="failed to get container status \"1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211\": rpc error: code = NotFound desc = could not find container \"1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211\": container with ID starting with 1c4ec4f33585aaec37a4425cb70f222be2d210cfc981e0d989106e0bd2a87211 not found: ID does not exist" Nov 01 14:45:46.713915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.713884 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume\") pod \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " Nov 01 14:45:46.714107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.713981 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f\" (UniqueName: \"kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f\") pod \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\" (UID: \"3f7fce6f-bebb-4816-82e0-1397baf4c4d8\") " Nov 01 14:45:46.714304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.714280 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume" pod "3f7fce6f-bebb-4816-82e0-1397baf4c4d8" (UID: "3f7fce6f-bebb-4816-82e0-1397baf4c4d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:46.714534 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.714518 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:46.714584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.714536 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w94b\" (UniqueName: \"kubernetes.io/projected/c26622bc-b8da-4d22-8f65-a297597266da-kube-api-access-4w94b\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:46.716011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.715995 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f" pod "3f7fce6f-bebb-4816-82e0-1397baf4c4d8" (UID: "3f7fce6f-bebb-4816-82e0-1397baf4c4d8"). InnerVolumeSpecName "kube-api-access-g275f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:46.816383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.816303 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g275f\" (UniqueName: \"kubernetes.io/projected/3f7fce6f-bebb-4816-82e0-1397baf4c4d8-kube-api-access-g275f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:46.847389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.847356 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7"] Nov 01 14:45:46.850608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.850588 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-71e39a-d4c7f84f7-smlx7"] Nov 01 14:45:46.852244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:46.852207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:47.004613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:47.004580 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:47.052130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:47.052101 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js"] Nov 01 14:45:47.055443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:47.055423 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-71e39a-5f597d487f-sh7js"] Nov 01 14:45:47.083620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:47.083541 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7fce6f-bebb-4816-82e0-1397baf4c4d8" path="/var/lib/kubelet/pods/3f7fce6f-bebb-4816-82e0-1397baf4c4d8/volumes" Nov 01 14:45:47.084101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:47.084079 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26622bc-b8da-4d22-8f65-a297597266da" path="/var/lib/kubelet/pods/c26622bc-b8da-4d22-8f65-a297597266da/volumes" Nov 01 14:45:47.438850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:47.438764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:49.359740 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:49.359703 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:50.932314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:50.932278 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:50.990241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:50.990174 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" podUID="40b711fd-a286-47cc-b661-090c7b7d0048" containerName="payload-processing" containerID="cri-o://cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c" gracePeriod=25 Nov 01 14:45:51.096284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.096210 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" podUID="042c5582-0f28-4bbd-8145-16489a946eb5" containerName="payload-pre-processing" containerID="cri-o://95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a" gracePeriod=25 Nov 01 14:45:51.293068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.293038 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:45:51.359527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.359504 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:45:51.392441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.392415 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume\") pod \"042c5582-0f28-4bbd-8145-16489a946eb5\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " Nov 01 14:45:51.392590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.392474 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb\" (UniqueName: \"kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb\") pod \"40b711fd-a286-47cc-b661-090c7b7d0048\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " Nov 01 14:45:51.392634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.392597 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p\" (UniqueName: \"kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p\") pod \"042c5582-0f28-4bbd-8145-16489a946eb5\" (UID: \"042c5582-0f28-4bbd-8145-16489a946eb5\") " Nov 01 14:45:51.392682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.392634 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume\") pod \"40b711fd-a286-47cc-b661-090c7b7d0048\" (UID: \"40b711fd-a286-47cc-b661-090c7b7d0048\") " Nov 01 14:45:51.392868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.392844 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume" pod "042c5582-0f28-4bbd-8145-16489a946eb5" (UID: "042c5582-0f28-4bbd-8145-16489a946eb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:51.393050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.393030 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume" pod "40b711fd-a286-47cc-b661-090c7b7d0048" (UID: "40b711fd-a286-47cc-b661-090c7b7d0048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:45:51.393212 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.393191 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/042c5582-0f28-4bbd-8145-16489a946eb5-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:51.393296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.393232 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40b711fd-a286-47cc-b661-090c7b7d0048-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:51.394654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.394632 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb" pod "40b711fd-a286-47cc-b661-090c7b7d0048" (UID: "40b711fd-a286-47cc-b661-090c7b7d0048"). InnerVolumeSpecName "kube-api-access-ng6pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:51.394753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.394728 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p" pod "042c5582-0f28-4bbd-8145-16489a946eb5" (UID: "042c5582-0f28-4bbd-8145-16489a946eb5"). InnerVolumeSpecName "kube-api-access-grk8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:45:51.494156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.494081 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6pb\" (UniqueName: \"kubernetes.io/projected/40b711fd-a286-47cc-b661-090c7b7d0048-kube-api-access-ng6pb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:51.494156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.494110 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grk8p\" (UniqueName: \"kubernetes.io/projected/042c5582-0f28-4bbd-8145-16489a946eb5-kube-api-access-grk8p\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:45:51.672766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.672735 5473 generic.go:356] "Generic (PLEG): container finished" podID="40b711fd-a286-47cc-b661-090c7b7d0048" containerID="cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c" exitCode=0 Nov 01 14:45:51.672960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.672781 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" Nov 01 14:45:51.672960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.672829 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" event={"ID":"40b711fd-a286-47cc-b661-090c7b7d0048","Type":"ContainerDied","Data":"cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c"} Nov 01 14:45:51.672960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.672887 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62" event={"ID":"40b711fd-a286-47cc-b661-090c7b7d0048","Type":"ContainerDied","Data":"9549421bb26755d68b83e43128a89de5bcb138db0374d9aa338abda5f07bf608"} Nov 01 14:45:51.672960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.672923 5473 scope.go:122] "RemoveContainer" containerID="cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c" Nov 01 14:45:51.674275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.674246 5473 generic.go:356] "Generic (PLEG): container finished" podID="042c5582-0f28-4bbd-8145-16489a946eb5" containerID="95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a" exitCode=0 Nov 01 14:45:51.674402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.674284 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" event={"ID":"042c5582-0f28-4bbd-8145-16489a946eb5","Type":"ContainerDied","Data":"95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a"} Nov 01 14:45:51.674402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.674302 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" Nov 01 14:45:51.674402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.674331 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq" event={"ID":"042c5582-0f28-4bbd-8145-16489a946eb5","Type":"ContainerDied","Data":"3ef282177c1f6b6041be55d228586b7de3e0dcb8fa8115e81676f4ff0f599114"} Nov 01 14:45:51.685126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.685110 5473 scope.go:122] "RemoveContainer" containerID="cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c" Nov 01 14:45:51.685402 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:51.685384 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c\": container with ID starting with cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c not found: ID does not exist" containerID="cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c" Nov 01 14:45:51.685484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.685414 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c"} err="failed to get container status \"cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c\": rpc error: code = NotFound desc = could not find container \"cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c\": container with ID starting with cb5ba26434e7eccb0e69d2ea4b3c725a0c33509dccb3f83bbea14956eb3f723c not found: ID does not exist" Nov 01 14:45:51.685484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.685436 5473 scope.go:122] "RemoveContainer" containerID="95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a" Nov 01 14:45:51.694808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.694789 5473 scope.go:122] "RemoveContainer" containerID="95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a" Nov 01 14:45:51.695058 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:45:51.695039 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a\": container with ID starting with 95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a not found: ID does not exist" containerID="95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a" Nov 01 14:45:51.695097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.695065 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a"} err="failed to get container status \"95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a\": rpc error: code = NotFound desc = could not find container \"95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a\": container with ID starting with 95a70b4080d1128698f27fb918ca00add957c803b013e69de9e086a9548bf26a not found: ID does not exist" Nov 01 14:45:51.838773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.838743 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62"] Nov 01 14:45:51.841948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.841922 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c9b0827c-77c9b9cdff-cwp62"] Nov 01 14:45:51.842889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.842867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:51.951855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.951824 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq"] Nov 01 14:45:51.955025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:51.955005 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c9b0827c-d85bbbdd7-zqpbq"] Nov 01 14:45:53.082627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:53.082595 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042c5582-0f28-4bbd-8145-16489a946eb5" path="/var/lib/kubelet/pods/042c5582-0f28-4bbd-8145-16489a946eb5/volumes" Nov 01 14:45:53.083074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:53.083058 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b711fd-a286-47cc-b661-090c7b7d0048" path="/var/lib/kubelet/pods/40b711fd-a286-47cc-b661-090c7b7d0048/volumes" Nov 01 14:45:54.142118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:54.142082 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:45:58.945959 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:45:58.945922 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:01.107715 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:01.107682 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:46:01.268175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:01.268145 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:02.121441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:02.121408 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:02.709997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:02.709960 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r"] Nov 01 14:46:02.742752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:02.742721 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:02.817429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:02.817348 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp"] Nov 01 14:46:02.964621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:02.964586 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:03.149990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:03.149906 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:03.400611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:03.400524 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:03.853469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:03.853425 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:05.552116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:05.552074 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:06.570684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:06.570653 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:07.404126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:07.404087 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh"] Nov 01 14:46:07.511755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:07.511723 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p"] Nov 01 14:46:07.726295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:07.721291 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" podUID="883ea872-216e-488f-a1a7-2eb1e6faf615" containerName="payload-processing" containerID="cri-o://72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df" gracePeriod=25 Nov 01 14:46:07.788587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:07.788561 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:07.830536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:07.829374 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" podUID="f152497f-f9d8-488a-9abd-4cdc59c79e0f" containerName="payload-pre-processing" containerID="cri-o://1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e" gracePeriod=25 Nov 01 14:46:07.964787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:07.964707 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:08.043499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.043474 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:46:08.151944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.151896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:08.170521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.170497 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:46:08.235200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.235175 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd\" (UniqueName: \"kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd\") pod \"883ea872-216e-488f-a1a7-2eb1e6faf615\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " Nov 01 14:46:08.235355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.235312 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume\") pod \"883ea872-216e-488f-a1a7-2eb1e6faf615\" (UID: \"883ea872-216e-488f-a1a7-2eb1e6faf615\") " Nov 01 14:46:08.235682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.235660 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume" pod "883ea872-216e-488f-a1a7-2eb1e6faf615" (UID: "883ea872-216e-488f-a1a7-2eb1e6faf615"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:08.235874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.235851 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/883ea872-216e-488f-a1a7-2eb1e6faf615-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:08.237329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.237304 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd" pod "883ea872-216e-488f-a1a7-2eb1e6faf615" (UID: "883ea872-216e-488f-a1a7-2eb1e6faf615"). InnerVolumeSpecName "kube-api-access-xgnqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:08.331133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.331106 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:08.336518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.336490 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84\" (UniqueName: \"kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84\") pod \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " Nov 01 14:46:08.336640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.336528 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume\") pod \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\" (UID: \"f152497f-f9d8-488a-9abd-4cdc59c79e0f\") " Nov 01 14:46:08.336898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.336878 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume" pod "f152497f-f9d8-488a-9abd-4cdc59c79e0f" (UID: "f152497f-f9d8-488a-9abd-4cdc59c79e0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:08.337067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.337053 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnqd\" (UniqueName: \"kubernetes.io/projected/883ea872-216e-488f-a1a7-2eb1e6faf615-kube-api-access-xgnqd\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:08.337113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.337069 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f152497f-f9d8-488a-9abd-4cdc59c79e0f-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:08.338715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.338695 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84" pod "f152497f-f9d8-488a-9abd-4cdc59c79e0f" (UID: "f152497f-f9d8-488a-9abd-4cdc59c79e0f"). InnerVolumeSpecName "kube-api-access-v2z84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:08.438164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.438132 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2z84\" (UniqueName: \"kubernetes.io/projected/f152497f-f9d8-488a-9abd-4cdc59c79e0f-kube-api-access-v2z84\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:08.748596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.748558 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:08.807484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.807447 5473 generic.go:356] "Generic (PLEG): container finished" podID="883ea872-216e-488f-a1a7-2eb1e6faf615" containerID="72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df" exitCode=0 Nov 01 14:46:08.808823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.808481 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" Nov 01 14:46:08.812336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.808502 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" event={"ID":"883ea872-216e-488f-a1a7-2eb1e6faf615","Type":"ContainerDied","Data":"72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df"} Nov 01 14:46:08.812806 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.812783 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r" event={"ID":"883ea872-216e-488f-a1a7-2eb1e6faf615","Type":"ContainerDied","Data":"8b995ea7712fe43e4b8efb0920ff3308fba034e33baa548904619fdfc0b924f2"} Nov 01 14:46:08.812982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.812930 5473 scope.go:122] "RemoveContainer" containerID="72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df" Nov 01 14:46:08.816628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.816587 5473 generic.go:356] "Generic (PLEG): container finished" podID="f152497f-f9d8-488a-9abd-4cdc59c79e0f" containerID="1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e" exitCode=0 Nov 01 14:46:08.816831 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.816813 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" event={"ID":"f152497f-f9d8-488a-9abd-4cdc59c79e0f","Type":"ContainerDied","Data":"1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e"} Nov 01 14:46:08.817020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.816986 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" event={"ID":"f152497f-f9d8-488a-9abd-4cdc59c79e0f","Type":"ContainerDied","Data":"756a631cb9f84d07ab47cfb7ae9259a786644bca7f959768837b18281072652c"} Nov 01 14:46:08.817252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.817209 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp" Nov 01 14:46:08.882867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.870413 5473 scope.go:122] "RemoveContainer" containerID="72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df" Nov 01 14:46:08.882867 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:08.870816 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df\": container with ID starting with 72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df not found: ID does not exist" containerID="72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df" Nov 01 14:46:08.882867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.870856 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df"} err="failed to get container status \"72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df\": rpc error: code = NotFound desc = could not find container \"72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df\": container with ID starting with 72da7cb331c6ae287a4e806f76b35febf25f92a763ae0f7b30643369dc3f06df not found: ID does not exist" Nov 01 14:46:08.882867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.870896 5473 scope.go:122] "RemoveContainer" containerID="1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e" Nov 01 14:46:08.885903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.885797 5473 scope.go:122] "RemoveContainer" containerID="1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e" Nov 01 14:46:08.886244 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:08.886188 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e\": container with ID starting with 1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e not found: ID does not exist" containerID="1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e" Nov 01 14:46:08.886327 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.886258 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e"} err="failed to get container status \"1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e\": rpc error: code = NotFound desc = could not find container \"1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e\": container with ID starting with 1eab04274a72acdac27fba1ed7751adb29261dfe0005dde01c4f422eb181557e not found: ID does not exist" Nov 01 14:46:09.004232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:08.998314 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp"] Nov 01 14:46:09.004232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:09.003797 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-25ce0b-6b69d6fd47-8cxfp"] Nov 01 14:46:09.096362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:09.096330 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f152497f-f9d8-488a-9abd-4cdc59c79e0f" path="/var/lib/kubelet/pods/f152497f-f9d8-488a-9abd-4cdc59c79e0f/volumes" Nov 01 14:46:09.110465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:09.110432 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r"] Nov 01 14:46:09.115791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:09.115767 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-25ce0b-86b9d6884c-lhh9r"] Nov 01 14:46:11.086057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:11.086026 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883ea872-216e-488f-a1a7-2eb1e6faf615" path="/var/lib/kubelet/pods/883ea872-216e-488f-a1a7-2eb1e6faf615/volumes" Nov 01 14:46:11.304739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:11.304700 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:11.475087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:11.475010 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:12.405430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.405394 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" podUID="dd8846e1-bd9d-49f5-842f-095f8e61b171" containerName="payload-processing" containerID="cri-o://03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d" gracePeriod=25 Nov 01 14:46:12.512425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.512392 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" podUID="d28f5824-7bc3-4fed-be98-8fe481d8f680" containerName="payload-pre-processing" containerID="cri-o://4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483" gracePeriod=25 Nov 01 14:46:12.676552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.676490 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:46:12.705476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.705450 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2\" (UniqueName: \"kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2\") pod \"dd8846e1-bd9d-49f5-842f-095f8e61b171\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " Nov 01 14:46:12.705624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.705493 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume\") pod \"dd8846e1-bd9d-49f5-842f-095f8e61b171\" (UID: \"dd8846e1-bd9d-49f5-842f-095f8e61b171\") " Nov 01 14:46:12.705957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.705930 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume" pod "dd8846e1-bd9d-49f5-842f-095f8e61b171" (UID: "dd8846e1-bd9d-49f5-842f-095f8e61b171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:12.706082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.706019 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8846e1-bd9d-49f5-842f-095f8e61b171-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:12.708546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.708515 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2" pod "dd8846e1-bd9d-49f5-842f-095f8e61b171" (UID: "dd8846e1-bd9d-49f5-842f-095f8e61b171"). InnerVolumeSpecName "kube-api-access-s2tb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:12.765373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.765347 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:46:12.806675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.806634 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume\") pod \"d28f5824-7bc3-4fed-be98-8fe481d8f680\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " Nov 01 14:46:12.806851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.806712 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v\" (UniqueName: \"kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v\") pod \"d28f5824-7bc3-4fed-be98-8fe481d8f680\" (UID: \"d28f5824-7bc3-4fed-be98-8fe481d8f680\") " Nov 01 14:46:12.806996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.806931 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume" pod "d28f5824-7bc3-4fed-be98-8fe481d8f680" (UID: "d28f5824-7bc3-4fed-be98-8fe481d8f680"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:12.807174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.807155 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2tb2\" (UniqueName: \"kubernetes.io/projected/dd8846e1-bd9d-49f5-842f-095f8e61b171-kube-api-access-s2tb2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:12.807272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.807174 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d28f5824-7bc3-4fed-be98-8fe481d8f680-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:12.808820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.808798 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v" pod "d28f5824-7bc3-4fed-be98-8fe481d8f680" (UID: "d28f5824-7bc3-4fed-be98-8fe481d8f680"). InnerVolumeSpecName "kube-api-access-n8f8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:12.908245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.908193 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8f8v\" (UniqueName: \"kubernetes.io/projected/d28f5824-7bc3-4fed-be98-8fe481d8f680-kube-api-access-n8f8v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:12.910715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.910684 5473 generic.go:356] "Generic (PLEG): container finished" podID="d28f5824-7bc3-4fed-be98-8fe481d8f680" containerID="4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483" exitCode=0 Nov 01 14:46:12.910829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.910722 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" event={"ID":"d28f5824-7bc3-4fed-be98-8fe481d8f680","Type":"ContainerDied","Data":"4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483"} Nov 01 14:46:12.910829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.910802 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" event={"ID":"d28f5824-7bc3-4fed-be98-8fe481d8f680","Type":"ContainerDied","Data":"38091e0c29f5ccae9fd3f197171f352d6c02d0a67a1b8f316d94150af2229498"} Nov 01 14:46:12.910829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.910739 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p" Nov 01 14:46:12.911004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.910832 5473 scope.go:122] "RemoveContainer" containerID="4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483" Nov 01 14:46:12.912570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.912542 5473 generic.go:356] "Generic (PLEG): container finished" podID="dd8846e1-bd9d-49f5-842f-095f8e61b171" containerID="03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d" exitCode=0 Nov 01 14:46:12.912701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.912589 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" Nov 01 14:46:12.912701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.912670 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" event={"ID":"dd8846e1-bd9d-49f5-842f-095f8e61b171","Type":"ContainerDied","Data":"03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d"} Nov 01 14:46:12.912798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.912725 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh" event={"ID":"dd8846e1-bd9d-49f5-842f-095f8e61b171","Type":"ContainerDied","Data":"922a648d8906928b01d12e06bbe6439a3e371795bff32b4897179c69d98daab8"} Nov 01 14:46:12.924683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.924663 5473 scope.go:122] "RemoveContainer" containerID="4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483" Nov 01 14:46:12.925449 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:12.925274 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483\": container with ID starting with 4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483 not found: ID does not exist" containerID="4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483" Nov 01 14:46:12.925449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.925307 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483"} err="failed to get container status \"4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483\": rpc error: code = NotFound desc = could not find container \"4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483\": container with ID starting with 4833fd4eca79f25e452feabf3a1ec74622fccc3cc85fded6f2950a83b36e3483 not found: ID does not exist" Nov 01 14:46:12.925449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.925331 5473 scope.go:122] "RemoveContainer" containerID="03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d" Nov 01 14:46:12.935887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.935867 5473 scope.go:122] "RemoveContainer" containerID="03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d" Nov 01 14:46:12.936251 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:12.936198 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d\": container with ID starting with 03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d not found: ID does not exist" containerID="03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d" Nov 01 14:46:12.936336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:12.936254 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d"} err="failed to get container status \"03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d\": rpc error: code = NotFound desc = could not find container \"03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d\": container with ID starting with 03554556e47e53a1e49f6c96390c4a1591fa599638a722d5ae16f8daef95ee5d not found: ID does not exist" Nov 01 14:46:13.095206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:13.095165 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p"] Nov 01 14:46:13.095206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:13.095231 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cbac83-5969d874c9-cx92p"] Nov 01 14:46:13.195503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:13.195416 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:13.207502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:13.207472 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh"] Nov 01 14:46:13.215965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:13.215939 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-cbac83-b7b8cb699-45zlh"] Nov 01 14:46:14.976460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:14.976421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:15.083276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:15.083246 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28f5824-7bc3-4fed-be98-8fe481d8f680" path="/var/lib/kubelet/pods/d28f5824-7bc3-4fed-be98-8fe481d8f680/volumes" Nov 01 14:46:15.083707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:15.083690 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8846e1-bd9d-49f5-842f-095f8e61b171" path="/var/lib/kubelet/pods/dd8846e1-bd9d-49f5-842f-095f8e61b171/volumes" Nov 01 14:46:17.034873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:17.034845 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:17.276499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:17.276465 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:18.549323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:18.549292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:19.029123 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.029092 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9"] Nov 01 14:46:19.029355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.029334 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" containerID="cri-o://5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62" gracePeriod=30 Nov 01 14:46:19.044411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.044382 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm"] Nov 01 14:46:19.044610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.044590 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" containerID="cri-o://cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66" gracePeriod=30 Nov 01 14:46:19.060973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.060951 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:19.634557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.634520 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:19.948586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:19.948507 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:21.012606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:21.012524 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:21.093318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:21.093283 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn"] Nov 01 14:46:21.143379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:21.143348 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff88d253-tqvzn"] Nov 01 14:46:21.200492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:21.200461 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:23.081715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:23.081682 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f4778b4-f0a1-41e8-a12a-13a4689d4583" path="/var/lib/kubelet/pods/5f4778b4-f0a1-41e8-a12a-13a4689d4583/volumes" Nov 01 14:46:24.312063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.311991 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:46:24.347530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.347504 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:46:24.475346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475312 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.475346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475350 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475383 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475430 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475476 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475535 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475562 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475603 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475711 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475731 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475809 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475895 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.475946 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.476005 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.476072 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.476102 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.476820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.476142 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.476175 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx\") pod \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\" (UID: \"b0b3e7a2-2b81-496d-a2dd-118fc29bb065\") " Nov 01 14:46:24.476820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.476320 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert\") pod \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\" (UID: \"cd8e867d-76f2-46e8-b270-65b5436b9ad0\") " Nov 01 14:46:24.478018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.477971 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.479843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.479804 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.480336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.479882 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480816 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480832 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480856 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480868 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480905 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480940 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480997 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:24.481023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.480994 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.481944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.481922 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.482054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.481984 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:24.485324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.482336 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:24.489206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.489170 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx" pod "b0b3e7a2-2b81-496d-a2dd-118fc29bb065" (UID: "b0b3e7a2-2b81-496d-a2dd-118fc29bb065"). InnerVolumeSpecName "kube-api-access-6wvsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:24.498886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.498859 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:46:24.499014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.498986 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:24.501459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.501425 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt" pod "cd8e867d-76f2-46e8-b270-65b5436b9ad0" (UID: "cd8e867d-76f2-46e8-b270-65b5436b9ad0"). InnerVolumeSpecName "kube-api-access-k7mzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:24.532282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.528882 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:24.583464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583380 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583410 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583424 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583437 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583449 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583461 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583475 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvsx\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-kube-api-access-6wvsx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583487 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583500 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583511 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583523 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583535 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7mzt\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-kube-api-access-k7mzt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583547 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583560 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583572 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0b3e7a2-2b81-496d-a2dd-118fc29bb065-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583584 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.583787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.583595 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd8e867d-76f2-46e8-b270-65b5436b9ad0-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:24.998792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.998710 5473 generic.go:356] "Generic (PLEG): container finished" podID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerID="5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62" exitCode=0 Nov 01 14:46:24.998792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.998780 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" Nov 01 14:46:24.998974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.998788 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" event={"ID":"cd8e867d-76f2-46e8-b270-65b5436b9ad0","Type":"ContainerDied","Data":"5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62"} Nov 01 14:46:24.998974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.998864 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9" event={"ID":"cd8e867d-76f2-46e8-b270-65b5436b9ad0","Type":"ContainerDied","Data":"b45453d66ef1de0fc02325ae7195c9c5602235c600d505a1e6e81a6c17cdadfc"} Nov 01 14:46:24.998974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:24.998886 5473 scope.go:122] "RemoveContainer" containerID="5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62" Nov 01 14:46:25.000459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.000436 5473 generic.go:356] "Generic (PLEG): container finished" podID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerID="cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66" exitCode=0 Nov 01 14:46:25.000595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.000489 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" Nov 01 14:46:25.000595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.000529 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" event={"ID":"b0b3e7a2-2b81-496d-a2dd-118fc29bb065","Type":"ContainerDied","Data":"cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66"} Nov 01 14:46:25.000595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.000571 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm" event={"ID":"b0b3e7a2-2b81-496d-a2dd-118fc29bb065","Type":"ContainerDied","Data":"990e0a1519d727e1310ee410559dccdb0aba5d2766d91796a63e25d6eebd9f72"} Nov 01 14:46:25.009931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.009907 5473 scope.go:122] "RemoveContainer" containerID="5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62" Nov 01 14:46:25.010268 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:25.010247 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62\": container with ID starting with 5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62 not found: ID does not exist" containerID="5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62" Nov 01 14:46:25.010363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.010277 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62"} err="failed to get container status \"5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62\": rpc error: code = NotFound desc = could not find container \"5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62\": container with ID starting with 5b74e0b8f09c53ba32e5800ebf73b4a34acdfadb93a5b5266fe3ea11a3691d62 not found: ID does not exist" Nov 01 14:46:25.010363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.010300 5473 scope.go:122] "RemoveContainer" containerID="cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66" Nov 01 14:46:25.019542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.019525 5473 scope.go:122] "RemoveContainer" containerID="cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66" Nov 01 14:46:25.019795 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:25.019775 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66\": container with ID starting with cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66 not found: ID does not exist" containerID="cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66" Nov 01 14:46:25.019857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.019805 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66"} err="failed to get container status \"cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66\": rpc error: code = NotFound desc = could not find container \"cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66\": container with ID starting with cccb0702a1e52df6d7dd7bf9c6ec8edbfbc1024450c596c4d8b87d61e02fcc66 not found: ID does not exist" Nov 01 14:46:25.165155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.165122 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9"] Nov 01 14:46:25.214900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.214867 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-dmlv9"] Nov 01 14:46:25.278317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.278286 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm"] Nov 01 14:46:25.281367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:25.281340 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26a0c1-openshift-default-76c87f685-g4smm"] Nov 01 14:46:26.139825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:26.139783 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:26.315855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:26.315825 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:27.081609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:27.081575 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" path="/var/lib/kubelet/pods/b0b3e7a2-2b81-496d-a2dd-118fc29bb065/volumes" Nov 01 14:46:27.082147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:27.082129 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" path="/var/lib/kubelet/pods/cd8e867d-76f2-46e8-b270-65b5436b9ad0/volumes" Nov 01 14:46:28.035421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:28.035392 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:28.596876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:28.596841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:29.269234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.269187 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:29.442553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.442520 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:29.827099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.827045 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst"] Nov 01 14:46:29.828164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828135 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c26622bc-b8da-4d22-8f65-a297597266da" containerName="payload-processing" Nov 01 14:46:29.828359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828343 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26622bc-b8da-4d22-8f65-a297597266da" containerName="payload-processing" Nov 01 14:46:29.828465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828454 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.828548 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828537 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.828644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828633 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:29.828723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828712 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:29.828807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828798 5473 cpu_manager.go:414] "containerMap: removing container" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.828883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828873 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.828968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.828959 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:29.829044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829034 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:29.829134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829125 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.829249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829201 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.829347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829335 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3f7fce6f-bebb-4816-82e0-1397baf4c4d8" containerName="payload-pre-processing" Nov 01 14:46:29.829470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829455 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7fce6f-bebb-4816-82e0-1397baf4c4d8" containerName="payload-pre-processing" Nov 01 14:46:29.829563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829554 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:29.829636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829626 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:29.829711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829701 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.829780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829771 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.829860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829851 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.829941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.829930 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.830024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830014 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:29.830099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830089 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:29.830177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830168 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" Nov 01 14:46:29.830274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830262 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" Nov 01 14:46:29.830379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830368 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:29.830458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830447 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:29.830541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830531 5473 cpu_manager.go:414] "containerMap: removing container" podUID="042c5582-0f28-4bbd-8145-16489a946eb5" containerName="payload-pre-processing" Nov 01 14:46:29.830617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830606 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042c5582-0f28-4bbd-8145-16489a946eb5" containerName="payload-pre-processing" Nov 01 14:46:29.830705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830695 5473 cpu_manager.go:414] "containerMap: removing container" podUID="40b711fd-a286-47cc-b661-090c7b7d0048" containerName="payload-processing" Nov 01 14:46:29.830804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830772 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b711fd-a286-47cc-b661-090c7b7d0048" containerName="payload-processing" Nov 01 14:46:29.830918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830907 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:29.831006 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.830995 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:29.831091 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831082 5473 cpu_manager.go:414] "containerMap: removing container" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.831167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831157 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.831272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831261 5473 cpu_manager.go:414] "containerMap: removing container" podUID="883ea872-216e-488f-a1a7-2eb1e6faf615" containerName="payload-processing" Nov 01 14:46:29.831361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831350 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883ea872-216e-488f-a1a7-2eb1e6faf615" containerName="payload-processing" Nov 01 14:46:29.831439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831430 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f152497f-f9d8-488a-9abd-4cdc59c79e0f" containerName="payload-pre-processing" Nov 01 14:46:29.831512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831502 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f152497f-f9d8-488a-9abd-4cdc59c79e0f" containerName="payload-pre-processing" Nov 01 14:46:29.831599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831589 5473 cpu_manager.go:414] "containerMap: removing container" podUID="dd8846e1-bd9d-49f5-842f-095f8e61b171" containerName="payload-processing" Nov 01 14:46:29.831669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831660 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8846e1-bd9d-49f5-842f-095f8e61b171" containerName="payload-processing" Nov 01 14:46:29.831757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831746 5473 cpu_manager.go:414] "containerMap: removing container" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" Nov 01 14:46:29.831829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831819 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" Nov 01 14:46:29.831912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831902 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d28f5824-7bc3-4fed-be98-8fe481d8f680" containerName="payload-pre-processing" Nov 01 14:46:29.832000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.831989 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28f5824-7bc3-4fed-be98-8fe481d8f680" containerName="payload-pre-processing" Nov 01 14:46:29.832081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832071 5473 cpu_manager.go:414] "containerMap: removing container" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.832162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832152 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.832265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832255 5473 cpu_manager.go:414] "containerMap: removing container" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.832350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832339 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.832431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832422 5473 cpu_manager.go:414] "containerMap: removing container" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.832493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832484 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.832718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832703 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.832818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832807 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="883ea872-216e-488f-a1a7-2eb1e6faf615" containerName="payload-processing" Nov 01 14:46:29.832892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832882 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f152497f-f9d8-488a-9abd-4cdc59c79e0f" containerName="payload-pre-processing" Nov 01 14:46:29.832976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.832965 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8846e1-bd9d-49f5-842f-095f8e61b171" containerName="payload-processing" Nov 01 14:46:29.833054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833046 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.833129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833119 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.833205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833196 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8e867d-76f2-46e8-b270-65b5436b9ad0" containerName="istio-proxy" Nov 01 14:46:29.833312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833301 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28f5824-7bc3-4fed-be98-8fe481d8f680" containerName="payload-pre-processing" Nov 01 14:46:29.833394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833385 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.833468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833459 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.833541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833532 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.833626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833616 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ed7b64-b4d7-4f09-af34-ac1b2737a6d0" containerName="cleanup" Nov 01 14:46:29.833707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833697 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:29.833786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833776 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:29.833862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833852 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:29.833938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.833928 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26622bc-b8da-4d22-8f65-a297597266da" containerName="payload-processing" Nov 01 14:46:29.834012 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834003 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.834084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834075 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:29.834151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834143 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.834247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834235 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7fce6f-bebb-4816-82e0-1397baf4c4d8" containerName="payload-pre-processing" Nov 01 14:46:29.834342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834332 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:29.834415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834406 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:29.834485 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834477 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:29.834561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834551 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:29.834638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834628 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b3e7a2-2b81-496d-a2dd-118fc29bb065" containerName="istio-proxy" Nov 01 14:46:29.834706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834697 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="042c5582-0f28-4bbd-8145-16489a946eb5" containerName="payload-pre-processing" Nov 01 14:46:29.834780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834771 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b711fd-a286-47cc-b661-090c7b7d0048" containerName="payload-processing" Nov 01 14:46:29.834858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.834848 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:29.840703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.840686 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:29.875461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.875424 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:29.875749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.875729 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpmdz\" (UniqueName: \"kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz\") pod \"payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:29.877747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.877721 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst"] Nov 01 14:46:29.919645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.919611 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-34d384\"" Nov 01 14:46:29.919645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.919634 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-34d384-dockercfg-wtjdw\"" Nov 01 14:46:29.977665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.977622 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:29.977665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.977658 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpmdz\" (UniqueName: \"kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz\") pod \"payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:29.978445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:29.978421 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:30.036336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:30.036301 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpmdz\" (UniqueName: \"kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz\") pod \"payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:30.168888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:30.168807 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:30.534512 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:46:30.534465 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4c07c9_7534_4872_808f_d5304f981dda.slice/crio-428f520df0051d7a3de92db9099ceed4c64c44be62221324812f9ae329784617 WatchSource:0}: Error finding container 428f520df0051d7a3de92db9099ceed4c64c44be62221324812f9ae329784617: Status 404 returned error can't find the container with id 428f520df0051d7a3de92db9099ceed4c64c44be62221324812f9ae329784617 Nov 01 14:46:30.537245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:30.537189 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst"] Nov 01 14:46:31.051442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:31.051402 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" event={"ID":"9b4c07c9-7534-4872-808f-d5304f981dda","Type":"ContainerStarted","Data":"24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d"} Nov 01 14:46:31.051605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:31.051470 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" event={"ID":"9b4c07c9-7534-4872-808f-d5304f981dda","Type":"ContainerStarted","Data":"428f520df0051d7a3de92db9099ceed4c64c44be62221324812f9ae329784617"} Nov 01 14:46:31.051605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:31.051500 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:31.108952 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:31.108922 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:46:31.166623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:31.166565 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" podStartSLOduration=2.166549878 podStartE2EDuration="2.166549878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:31.164429084 +0000 UTC m=+1730.222580935" watchObservedRunningTime="2026-11-01 14:46:31.166549878 +0000 UTC m=+1730.224701727" Nov 01 14:46:31.292697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:31.292657 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:36.308604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:36.308567 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:42.062118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:42.062087 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:46:43.261660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:43.261629 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:43.680820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:43.680741 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg"] Nov 01 14:46:43.680969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:43.680947 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" containerID="cri-o://2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc" gracePeriod=30 Nov 01 14:46:44.861784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:44.861749 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:48.299509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:48.299440 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:48.942388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:48.942333 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:46:49.100891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.100863 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.100916 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.100940 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.100983 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101017 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101040 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101090 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101125 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101237 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm\") pod \"ea0008a8-5b95-42e5-b0d3-956155648064\" (UID: \"ea0008a8-5b95-42e5-b0d3-956155648064\") " Nov 01 14:46:49.101364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101237 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:49.101584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101415 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:49.101584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101439 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:46:49.101720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101688 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:49.101913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101847 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.101913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101863 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:49.101913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101874 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.101913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101891 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0008a8-5b95-42e5-b0d3-956155648064-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.101913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.101905 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.103339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.103308 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:46:49.103748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.103728 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:46:49.103857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.103835 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "kube-api-access-4t9zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:49.103904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.103860 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token" pod "ea0008a8-5b95-42e5-b0d3-956155648064" (UID: "ea0008a8-5b95-42e5-b0d3-956155648064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:46:49.179122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.179082 5473 generic.go:356] "Generic (PLEG): container finished" podID="ea0008a8-5b95-42e5-b0d3-956155648064" containerID="2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc" exitCode=0 Nov 01 14:46:49.179338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.179151 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" Nov 01 14:46:49.179338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.179268 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" event={"ID":"ea0008a8-5b95-42e5-b0d3-956155648064","Type":"ContainerDied","Data":"2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc"} Nov 01 14:46:49.179338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.179316 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg" event={"ID":"ea0008a8-5b95-42e5-b0d3-956155648064","Type":"ContainerDied","Data":"556073942b27db107e6f54b9078288ea14f0044d861b8f3c47c248d36cd83924"} Nov 01 14:46:49.179528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.179344 5473 scope.go:122] "RemoveContainer" containerID="2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc" Nov 01 14:46:49.191922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.191896 5473 scope.go:122] "RemoveContainer" containerID="2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc" Nov 01 14:46:49.192226 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:46:49.192194 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc\": container with ID starting with 2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc not found: ID does not exist" containerID="2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc" Nov 01 14:46:49.192333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.192240 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc"} err="failed to get container status \"2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc\": rpc error: code = NotFound desc = could not find container \"2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc\": container with ID starting with 2d8d6277e6fc44a5d98eb2a46f443472c4b4cfc22fe4d20cd23ad2fc3d289cbc not found: ID does not exist" Nov 01 14:46:49.202988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.202941 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.202988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.202965 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0008a8-5b95-42e5-b0d3-956155648064-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.202988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.202976 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.202988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.202985 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0008a8-5b95-42e5-b0d3-956155648064-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.203167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.202994 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9zm\" (UniqueName: \"kubernetes.io/projected/ea0008a8-5b95-42e5-b0d3-956155648064-kube-api-access-4t9zm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:46:49.227777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.227757 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:49.347864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.347837 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg"] Nov 01 14:46:49.350729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.350710 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c9b0827c-openshift-default-597b84c65c-cbvvg"] Nov 01 14:46:49.705089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705059 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z"] Nov 01 14:46:49.705650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705636 5473 cpu_manager.go:414] "containerMap: removing container" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705651 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058a1791-1bce-4b5b-9248-a6a24cb36993" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705663 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705669 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705685 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705691 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705698 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:49.705704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705702 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7093fb3-a8c6-4f31-a246-495885c4450e" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705710 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705715 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7906b7-1585-49a0-90f4-ed0adc65b92d" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705723 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705728 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705735 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705739 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705813 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f712cd5-7799-4d23-8f09-67fb58da9a70" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705825 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc1c5ac-3e82-48a8-8616-b6c14f78e5f2" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705832 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e198cc6e-50c2-4994-9ce6-1c40f3d32b18" containerName="cleanup" Nov 01 14:46:49.705919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.705839 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" containerName="istio-proxy" Nov 01 14:46:49.710817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.710798 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.764241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.763586 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7c335a-openshift-default-dockercfg-ggdm8\"" Nov 01 14:46:49.793250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.791258 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z"] Nov 01 14:46:49.862561 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.862521 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.866994 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.867070 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.867135 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.867168 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.867196 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.867243 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.867451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.867303 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwhv4\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.868384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.868341 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.868812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.868643 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970459 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970532 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970631 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970681 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970741 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970771 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.970804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970799 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971053 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970826 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971053 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.970892 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwhv4\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.971040 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.971098 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.971163 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.971387 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.971563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.971546 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.973147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.973122 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:49.973307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:49.973290 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:50.027920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.027886 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:50.028066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.027982 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwhv4\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4\") pod \"e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:50.029894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.029871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:50.325116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.325085 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:50.337022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.336995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:50.742490 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:46:50.742456 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a714983_4771_43a1_b823_129a908d4254.slice/crio-a6c132777700c20a3ee486705fdf02cadc2f13e64c219b21ba4d9dd2006638a2 WatchSource:0}: Error finding container a6c132777700c20a3ee486705fdf02cadc2f13e64c219b21ba4d9dd2006638a2: Status 404 returned error can't find the container with id a6c132777700c20a3ee486705fdf02cadc2f13e64c219b21ba4d9dd2006638a2 Nov 01 14:46:50.744913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.744876 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:46:50.745013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.744948 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:46:50.745013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.744977 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:46:50.745510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.745425 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z"] Nov 01 14:46:50.795893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.795863 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:50.955201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:50.955165 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:51.083501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.083470 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0008a8-5b95-42e5-b0d3-956155648064" path="/var/lib/kubelet/pods/ea0008a8-5b95-42e5-b0d3-956155648064/volumes" Nov 01 14:46:51.194367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.194328 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" event={"ID":"2a714983-4771-43a1-b823-129a908d4254","Type":"ContainerStarted","Data":"9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133"} Nov 01 14:46:51.194509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.194384 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" event={"ID":"2a714983-4771-43a1-b823-129a908d4254","Type":"ContainerStarted","Data":"a6c132777700c20a3ee486705fdf02cadc2f13e64c219b21ba4d9dd2006638a2"} Nov 01 14:46:51.315726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.315677 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podStartSLOduration=2.315663104 podStartE2EDuration="2.315663104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:51.312684532 +0000 UTC m=+1750.370836364" watchObservedRunningTime="2026-11-01 14:46:51.315663104 +0000 UTC m=+1750.373814953" Nov 01 14:46:51.325248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.325224 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:51.330826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.330803 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:51.993851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:51.993819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:52.158609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:52.158568 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:52.247517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:52.247402 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:52.248487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:52.248467 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:46:52.324770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:52.324739 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:52.585911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:52.585878 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:54.057136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:54.057105 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:54.325051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:54.324967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:55.470397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:55.470364 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:57.446278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.446246 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:57.657395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.657362 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:57.714995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.714923 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79"] Nov 01 14:46:57.719189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.719171 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.772769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.772739 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79"] Nov 01 14:46:57.790799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.790765 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.790977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.790929 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.791051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.791023 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t28h5\" (UniqueName: \"kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.817473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.817450 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:46:57.817473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.817450 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:46:57.872849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.872821 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr"] Nov 01 14:46:57.876842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.876826 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:57.892072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.892041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.892198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.892095 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.892198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.892137 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t28h5\" (UniqueName: \"kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.892706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.892686 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.894439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.894416 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:57.929211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.929148 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7c335a\"" Nov 01 14:46:57.929417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.929346 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7c335a-dockercfg-bc8hl\"" Nov 01 14:46:57.931248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.931189 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr"] Nov 01 14:46:57.994325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:57.992686 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t28h5\" (UniqueName: \"kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5\") pod \"maas-api-revoke-sub-44ace39b-p6b79\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:58.003154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.001644 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.003154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.001934 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqsv\" (UniqueName: \"kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.034266 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.033461 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:46:58.102858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.102826 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.103034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.103016 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqsv\" (UniqueName: \"kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.103717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.103687 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.160183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.160144 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqsv\" (UniqueName: \"kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv\") pod \"payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.190735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.190698 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:58.430968 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:46:58.430935 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb6143d1_38fa_4f4e_b2b7_f996f2135ccb.slice/crio-d9e9a16eed2d81cff72544381cce29646a0b5ca22ac24bccf90f4f926e906ac1 WatchSource:0}: Error finding container d9e9a16eed2d81cff72544381cce29646a0b5ca22ac24bccf90f4f926e906ac1: Status 404 returned error can't find the container with id d9e9a16eed2d81cff72544381cce29646a0b5ca22ac24bccf90f4f926e906ac1 Nov 01 14:46:58.434370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.434346 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79"] Nov 01 14:46:58.557538 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:46:58.557512 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d5cd57e_fb93_498e_8c71_90e54883e644.slice/crio-9b7328588d7e4ff3eea0d8f658c9ce37dc38f415394f441e2cc19cd02ba6f36b WatchSource:0}: Error finding container 9b7328588d7e4ff3eea0d8f658c9ce37dc38f415394f441e2cc19cd02ba6f36b: Status 404 returned error can't find the container with id 9b7328588d7e4ff3eea0d8f658c9ce37dc38f415394f441e2cc19cd02ba6f36b Nov 01 14:46:58.560669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.560650 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr"] Nov 01 14:46:58.891411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:58.891329 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:59.296673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.296342 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" event={"ID":"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb","Type":"ContainerStarted","Data":"52fb0ef4ada4c3c432b71487bf6cd26d73ab0263348dbdcb0421de13b13c3a98"} Nov 01 14:46:59.296673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.296429 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" event={"ID":"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb","Type":"ContainerStarted","Data":"d9e9a16eed2d81cff72544381cce29646a0b5ca22ac24bccf90f4f926e906ac1"} Nov 01 14:46:59.298136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.298110 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" event={"ID":"7d5cd57e-fb93-498e-8c71-90e54883e644","Type":"ContainerStarted","Data":"aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432"} Nov 01 14:46:59.298279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.298156 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" event={"ID":"7d5cd57e-fb93-498e-8c71-90e54883e644","Type":"ContainerStarted","Data":"9b7328588d7e4ff3eea0d8f658c9ce37dc38f415394f441e2cc19cd02ba6f36b"} Nov 01 14:46:59.298279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.298177 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:46:59.343826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.343800 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:46:59.415333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.415270 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" podStartSLOduration=2.415251252 podStartE2EDuration="2.415251252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:59.410448041 +0000 UTC m=+1758.468599928" watchObservedRunningTime="2026-11-01 14:46:59.415251252 +0000 UTC m=+1758.473403103" Nov 01 14:46:59.526007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:46:59.525791 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" podStartSLOduration=2.525771535 podStartE2EDuration="2.525771535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:46:59.523740008 +0000 UTC m=+1758.581891860" watchObservedRunningTime="2026-11-01 14:46:59.525771535 +0000 UTC m=+1758.583923387" Nov 01 14:47:00.305984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:00.305950 5473 generic.go:356] "Generic (PLEG): container finished" podID="eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" containerID="52fb0ef4ada4c3c432b71487bf6cd26d73ab0263348dbdcb0421de13b13c3a98" exitCode=0 Nov 01 14:47:00.306483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:00.306034 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" event={"ID":"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb","Type":"ContainerDied","Data":"52fb0ef4ada4c3c432b71487bf6cd26d73ab0263348dbdcb0421de13b13c3a98"} Nov 01 14:47:01.108680 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:47:01.108648 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:47:01.463636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.463608 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:47:01.468507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.468477 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5\" (UniqueName: \"kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5\") pod \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " Nov 01 14:47:01.468650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.468550 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca\") pod \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " Nov 01 14:47:01.468717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.468658 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca\") pod \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\" (UID: \"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb\") " Nov 01 14:47:01.468904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.468883 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca" pod "eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" (UID: "eb6143d1-38fa-4f4e-b2b7-f996f2135ccb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:47:01.469335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.469309 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:01.470586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.470562 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca" pod "eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" (UID: "eb6143d1-38fa-4f4e-b2b7-f996f2135ccb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:47:01.470956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.470926 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5" pod "eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" (UID: "eb6143d1-38fa-4f4e-b2b7-f996f2135ccb"). InnerVolumeSpecName "kube-api-access-t28h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:47:01.570191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.570158 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t28h5\" (UniqueName: \"kubernetes.io/projected/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-kube-api-access-t28h5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:01.570191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.570183 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:01.825800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:01.825761 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:02.323657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:02.323618 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" event={"ID":"eb6143d1-38fa-4f4e-b2b7-f996f2135ccb","Type":"ContainerDied","Data":"d9e9a16eed2d81cff72544381cce29646a0b5ca22ac24bccf90f4f926e906ac1"} Nov 01 14:47:02.323657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:02.323655 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79" Nov 01 14:47:02.323657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:02.323664 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9e9a16eed2d81cff72544381cce29646a0b5ca22ac24bccf90f4f926e906ac1" Nov 01 14:47:06.205121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:06.205027 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:06.392725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:06.392623 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:06.751239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:06.747971 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:08.438489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:08.438454 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:10.309470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:10.309443 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:47:10.896792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:10.896757 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:14.051016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:14.050976 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:15.416749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:15.415391 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:19.906852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:19.906811 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:20.703877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:20.703840 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:23.049439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:23.049404 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz"] Nov 01 14:47:23.099284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:23.099256 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47e6a230-5k9pz"] Nov 01 14:47:24.316379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.316345 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h"] Nov 01 14:47:24.316912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.316893 5473 cpu_manager.go:414] "containerMap: removing container" podUID="eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" containerName="revoke-keys" Nov 01 14:47:24.316912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.316908 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" containerName="revoke-keys" Nov 01 14:47:24.317058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.317048 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" containerName="revoke-keys" Nov 01 14:47:24.321479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.321459 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.410698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.410655 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qmq\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.410869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.410788 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.410869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.410830 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.411000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.410972 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.411066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.411005 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.411183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.411164 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.411316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.411298 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.411389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.411341 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.411453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.411416 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.444703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.444674 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:24.513693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.513650 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h"] Nov 01 14:47:24.515618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.515583 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.515838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.515632 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.515838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.515727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.515838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.515751 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.515883 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.515956 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.516003 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.516038 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.516095 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qmq\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.516485 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.516823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.516797 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.525355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.517159 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.525355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.517430 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.525355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.517763 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.525355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.519151 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.525355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.519481 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.578553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.578489 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qmq\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.581167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.581142 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pbj6h\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:24.635585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:24.635559 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:25.050737 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:47:25.050701 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51ee30d_db82_4b0a_afb8_881486f4f982.slice/crio-b6449683668b3023e0b22d955ca26b8dbb84535e6ec5594b66ef49e10a98b448 WatchSource:0}: Error finding container b6449683668b3023e0b22d955ca26b8dbb84535e6ec5594b66ef49e10a98b448: Status 404 returned error can't find the container with id b6449683668b3023e0b22d955ca26b8dbb84535e6ec5594b66ef49e10a98b448 Nov 01 14:47:25.053650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.053612 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:47:25.053748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.053697 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:47:25.053748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.053739 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:47:25.054829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.054810 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h"] Nov 01 14:47:25.083947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.083923 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5134cc67-8517-441e-ad1b-313e231a372d" path="/var/lib/kubelet/pods/5134cc67-8517-441e-ad1b-313e231a372d/volumes" Nov 01 14:47:25.565729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.565692 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" event={"ID":"b51ee30d-db82-4b0a-afb8-881486f4f982","Type":"ContainerStarted","Data":"2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11"} Nov 01 14:47:25.566127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.565744 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" event={"ID":"b51ee30d-db82-4b0a-afb8-881486f4f982","Type":"ContainerStarted","Data":"b6449683668b3023e0b22d955ca26b8dbb84535e6ec5594b66ef49e10a98b448"} Nov 01 14:47:25.636528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.636493 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:25.753425 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:25.753354 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podStartSLOduration=1.753330601 podStartE2EDuration="1.753330601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:47:25.741332394 +0000 UTC m=+1784.799484245" watchObservedRunningTime="2026-11-01 14:47:25.753330601 +0000 UTC m=+1784.811482453" Nov 01 14:47:26.549270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:26.546292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:26.650017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:26.649980 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pbj6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 14:47:26.650451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:26.650035 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 14:47:27.646239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:27.643573 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pbj6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 14:47:27.646239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:27.643632 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 14:47:28.644249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:28.643614 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pbj6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 14:47:28.644249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:28.643676 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 14:47:29.646241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:29.643531 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pbj6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 14:47:29.646241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:29.643593 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 14:47:30.649238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:30.648547 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pbj6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 14:47:30.649238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:30.648618 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 14:47:30.754994 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:30.754959 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:30.916958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:30.916873 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:31.081106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:31.081067 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:31.108053 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:47:31.108020 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:47:31.636671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:31.636628 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pbj6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 01 14:47:31.636850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:31.636695 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 01 14:47:33.462066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:33.462032 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:33.462546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:33.462494 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:33.463726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:33.463704 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:47:35.410458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:35.410422 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:39.911239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:39.889532 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:40.151267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:40.150997 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:40.376429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:40.376394 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:41.022958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:41.022927 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:41.335636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:41.335531 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:41.710392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:41.710314 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:42.466963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:42.466932 5473 scope.go:122] "RemoveContainer" containerID="cdff265425c0f79cde6dd22738dac3a1c2c1aa1ebbadf3f3fe899ba539fb6503" Nov 01 14:47:42.478151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:42.478126 5473 scope.go:122] "RemoveContainer" containerID="8a9c45f907484f2bd124f918f5db5bdd289bf51e1c7b6f6d0f731f2bb191b544" Nov 01 14:47:43.032163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.032132 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:43.295101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.294990 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:43.713237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.713139 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:47:43.713814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.713778 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h"] Nov 01 14:47:43.714030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.713994 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" containerID="cri-o://687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f" gracePeriod=30 Nov 01 14:47:43.715546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.715527 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:47:43.737842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:43.737800 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:46.648334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:46.648288 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:47.074007 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:47.073972 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:47:47.074201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:47.074038 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:47:47.560193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:47.560161 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:48.988558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:48.988505 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:47:49.089711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.089670 5473 generic.go:356] "Generic (PLEG): container finished" podID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerID="687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f" exitCode=0 Nov 01 14:47:49.089914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.089772 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" Nov 01 14:47:49.089983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.089933 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090011 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090030 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090055 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090301 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090151 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090301 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090275 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090317 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090346 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn\") pod \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\" (UID: \"0b77a03f-bf88-4ea5-abfa-ee50285637b1\") " Nov 01 14:47:49.090512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090393 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:47:49.090512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090400 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:47:49.090512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090415 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:47:49.090667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090588 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:47:49.090745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090725 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:47:49.090923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090899 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.090923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090925 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.091121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090939 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.091121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090952 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.091121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.090965 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.092672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.092649 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:47:49.092816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.092799 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "kube-api-access-pkhvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:47:49.093380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.093360 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:47:49.094536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.094511 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo" pod "0b77a03f-bf88-4ea5-abfa-ee50285637b1" (UID: "0b77a03f-bf88-4ea5-abfa-ee50285637b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:47:49.114233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.114197 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" event={"ID":"0b77a03f-bf88-4ea5-abfa-ee50285637b1","Type":"ContainerDied","Data":"687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f"} Nov 01 14:47:49.114326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.114270 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h" event={"ID":"0b77a03f-bf88-4ea5-abfa-ee50285637b1","Type":"ContainerDied","Data":"e95794dddda5c6404a2d4d27a1b7fc3c9274cfd646ae47a389944c601f66d5f3"} Nov 01 14:47:49.114326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.114304 5473 scope.go:122] "RemoveContainer" containerID="687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f" Nov 01 14:47:49.123749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.123734 5473 scope.go:122] "RemoveContainer" containerID="7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba" Nov 01 14:47:49.132708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.132685 5473 scope.go:122] "RemoveContainer" containerID="687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f" Nov 01 14:47:49.132991 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:47:49.132972 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f\": container with ID starting with 687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f not found: ID does not exist" containerID="687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f" Nov 01 14:47:49.133042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.133000 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f"} err="failed to get container status \"687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f\": rpc error: code = NotFound desc = could not find container \"687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f\": container with ID starting with 687fa390ed586175209ce77755026f94f56ce6997254ad6b1e170e13aaf3544f not found: ID does not exist" Nov 01 14:47:49.133042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.133018 5473 scope.go:122] "RemoveContainer" containerID="7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba" Nov 01 14:47:49.133273 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:47:49.133255 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba\": container with ID starting with 7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba not found: ID does not exist" containerID="7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba" Nov 01 14:47:49.133326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.133279 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba"} err="failed to get container status \"7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba\": rpc error: code = NotFound desc = could not find container \"7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba\": container with ID starting with 7f2d4eaea14bdfa5ecbe93f75668a57346c2c335ea6e95692df62f509b518bba not found: ID does not exist" Nov 01 14:47:49.192063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.192018 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.192063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.192053 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.192338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.192071 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkhvn\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-kube-api-access-pkhvn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.192338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.192086 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b77a03f-bf88-4ea5-abfa-ee50285637b1-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:47:49.511886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.511851 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h"] Nov 01 14:47:49.514786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:49.514765 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-18a879-openshift-default-549dcd5875-ts55h"] Nov 01 14:47:51.081899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:51.081811 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" path="/var/lib/kubelet/pods/0b77a03f-bf88-4ea5-abfa-ee50285637b1/volumes" Nov 01 14:47:52.552448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:52.552413 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:52.742928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:52.742895 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:52.906323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:52.906207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:53.089582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:53.089554 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:53.514798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:53.514764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:53.692268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:53.692209 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:54.840181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:54.840149 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:47:57.611198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:47:57.611155 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:00.410798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:00.410764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:01.120712 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:48:01.120678 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:48:03.841428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:03.841393 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:06.644430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.644393 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf"] Nov 01 14:48:06.644958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.644944 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" Nov 01 14:48:06.645002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.644959 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" Nov 01 14:48:06.645002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.644978 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" Nov 01 14:48:06.645002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.644983 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" Nov 01 14:48:06.645108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.645103 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" Nov 01 14:48:06.645142 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.645112 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b77a03f-bf88-4ea5-abfa-ee50285637b1" containerName="istio-proxy" Nov 01 14:48:06.648417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.648402 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.700638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.700614 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-34d384-openshift-default-dockercfg-7trgk\"" Nov 01 14:48:06.756603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.756567 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf"] Nov 01 14:48:06.798900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.798868 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.798928 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799008 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799038 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799146 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799238 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799287 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpwt\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799358 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.799409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.799384 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.900811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900735 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.900811 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900767 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900816 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900862 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900883 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900914 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.900956 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901027 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901080 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpwt\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901264 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901292 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901371 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901521 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.901863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.901837 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.903141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.903120 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.903457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.903440 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.958415 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.958388 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.958672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.958649 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpwt\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt\") pod \"e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:06.962346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:06.962326 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:07.378986 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:48:07.378957 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9d24b2_93c6_4e08_b2a0_f7841567a71a.slice/crio-8dbea7f99d985f53b0ee9ecfe8ee8eadd4016e75a2345e9d0dcd6504e69f512d WatchSource:0}: Error finding container 8dbea7f99d985f53b0ee9ecfe8ee8eadd4016e75a2345e9d0dcd6504e69f512d: Status 404 returned error can't find the container with id 8dbea7f99d985f53b0ee9ecfe8ee8eadd4016e75a2345e9d0dcd6504e69f512d Nov 01 14:48:07.381706 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:07.381666 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:48:07.381810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:07.381750 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:48:07.381810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:07.381790 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:48:07.383129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:07.383108 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf"] Nov 01 14:48:08.257085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.257039 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" event={"ID":"7e9d24b2-93c6-4e08-b2a0-f7841567a71a","Type":"ContainerStarted","Data":"92193297149cd4434e5ac669201f1b8abbdd6415c2cd182a9418fb39e4af5f79"} Nov 01 14:48:08.257587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.257138 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" event={"ID":"7e9d24b2-93c6-4e08-b2a0-f7841567a71a","Type":"ContainerStarted","Data":"8dbea7f99d985f53b0ee9ecfe8ee8eadd4016e75a2345e9d0dcd6504e69f512d"} Nov 01 14:48:08.430276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.426893 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podStartSLOduration=2.426872761 podStartE2EDuration="2.426872761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:48:08.423590437 +0000 UTC m=+1827.481742300" watchObservedRunningTime="2026-11-01 14:48:08.426872761 +0000 UTC m=+1827.485024614" Nov 01 14:48:08.768244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.768181 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:08.962932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.962900 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:08.964349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.964323 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 01 14:48:08.964456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:08.964381 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 01 14:48:09.963537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:09.963506 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 01 14:48:09.963950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:09.963576 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 01 14:48:10.963235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:10.963177 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 01 14:48:10.963490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:10.963260 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 01 14:48:11.962925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:11.962893 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 01 14:48:11.963335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:11.962955 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 01 14:48:12.963010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:12.962976 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 01 14:48:12.963418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:12.963031 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 01 14:48:13.541556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:13.541505 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:13.967750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:13.967667 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:13.968210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:13.968193 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:13.968929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:13.968909 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:48:15.154943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:15.154910 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:15.457874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:15.457790 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:15.621032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:15.620996 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:18.463539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:18.463497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:22.942250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:22.938515 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:27.508534 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:27.508497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:30.420232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:30.420193 5473 generic.go:356] "Generic (PLEG): container finished" podID="2a714983-4771-43a1-b823-129a908d4254" containerID="9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133" exitCode=0 Nov 01 14:48:30.420672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:30.420269 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" event={"ID":"2a714983-4771-43a1-b823-129a908d4254","Type":"ContainerDied","Data":"9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133"} Nov 01 14:48:30.420672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:30.420617 5473 scope.go:122] "RemoveContainer" containerID="9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133" Nov 01 14:48:30.422019 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:30.421992 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:48:30.422115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:30.422055 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:48:30.422115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:30.422091 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:48:31.057140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:31.057046 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5"] Nov 01 14:48:31.057418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:31.057389 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15da1799-c8jn5"] Nov 01 14:48:31.082229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:31.082182 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ab0917-810c-476c-a535-9a031d74c629" path="/var/lib/kubelet/pods/c4ab0917-810c-476c-a535-9a031d74c629/volumes" Nov 01 14:48:31.107567 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:48:31.107537 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:48:31.432541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:31.432239 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" event={"ID":"2a714983-4771-43a1-b823-129a908d4254","Type":"ContainerStarted","Data":"69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc"} Nov 01 14:48:32.347586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:32.347553 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:48:32.347932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:32.347909 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 14:48:32.348112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:32.348087 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 14:48:33.337241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:33.330537 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 14:48:33.337241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:33.330597 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 14:48:33.503585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:33.503556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:34.348246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:34.338535 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 14:48:34.348246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:34.338606 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 14:48:35.349606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:35.349552 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 14:48:35.350124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:35.349610 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 14:48:36.012042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:36.012005 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl"] Nov 01 14:48:36.012247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:36.012206 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" containerID="cri-o://56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0" gracePeriod=30 Nov 01 14:48:36.343324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:36.337520 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 01 14:48:36.343324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:36.337590 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 01 14:48:37.227010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:37.226975 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:48:37.227410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:37.227047 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:48:37.263056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:37.263018 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:48:37.336849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:37.336396 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:48:37.490613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:37.490542 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:48:38.044732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:38.044694 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:39.341628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.341592 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk"] Nov 01 14:48:39.345926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.345907 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.397272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.397245 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk"] Nov 01 14:48:39.422741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.422720 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:48:39.443757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.443734 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:48:39.492603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.492579 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fpr\" (UniqueName: \"kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.492715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.492636 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.492813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.492793 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.594346 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.594268 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.594481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.594352 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fpr\" (UniqueName: \"kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.594481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.594406 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.595173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.595142 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.596721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.596695 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.651516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.651489 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fpr\" (UniqueName: \"kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr\") pod \"maas-api-revoke-sub-4db15f32-wdsxk\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:39.659452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:39.659436 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:40.029057 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:48:40.029028 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf834c040_a487_468d_9806_e517879c1675.slice/crio-d7494a426e1bb13eb44b2183059b1627a587315fc9bd1d3665b425f0feb53c32 WatchSource:0}: Error finding container d7494a426e1bb13eb44b2183059b1627a587315fc9bd1d3665b425f0feb53c32: Status 404 returned error can't find the container with id d7494a426e1bb13eb44b2183059b1627a587315fc9bd1d3665b425f0feb53c32 Nov 01 14:48:40.034993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:40.034969 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk"] Nov 01 14:48:40.512708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:40.512673 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" event={"ID":"f834c040-a487-468d-9806-e517879c1675","Type":"ContainerStarted","Data":"1e40cf29b994f0a638a0a21fb9d6d71485f4a970f7e7566786d466c6ca68a552"} Nov 01 14:48:40.513093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:40.512725 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" event={"ID":"f834c040-a487-468d-9806-e517879c1675","Type":"ContainerStarted","Data":"d7494a426e1bb13eb44b2183059b1627a587315fc9bd1d3665b425f0feb53c32"} Nov 01 14:48:40.644665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:40.644570 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" podStartSLOduration=1.644552702 podStartE2EDuration="1.644552702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:48:40.640896355 +0000 UTC m=+1859.699048205" watchObservedRunningTime="2026-11-01 14:48:40.644552702 +0000 UTC m=+1859.702704553" Nov 01 14:48:41.289108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.289055 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:48:41.417246 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417186 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.417414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417287 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.417414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417327 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.417414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417388 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.417594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417445 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.417594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417480 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.417991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417621 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:48:41.417991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417646 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:48:41.418181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417853 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:48:41.418181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417881 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:48:41.418181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.417998 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.418181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418052 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.418181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418125 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy\") pod \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\" (UID: \"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082\") " Nov 01 14:48:41.418510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418377 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:48:41.418868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418772 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.418868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418802 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.418868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418817 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.418868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418830 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.418868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.418845 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.419808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.419783 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "kube-api-access-bgtl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:48:41.419910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.419846 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:48:41.420233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.420202 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:48:41.420589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.420570 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo" pod "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" (UID: "3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:48:41.520514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.520487 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.520514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.520516 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.520992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.520527 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.520992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.520536 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgtl5\" (UniqueName: \"kubernetes.io/projected/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082-kube-api-access-bgtl5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:41.521125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.521098 5473 generic.go:356] "Generic (PLEG): container finished" podID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerID="56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0" exitCode=0 Nov 01 14:48:41.521257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.521130 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" event={"ID":"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082","Type":"ContainerDied","Data":"56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0"} Nov 01 14:48:41.521257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.521169 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" Nov 01 14:48:41.521257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.521190 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl" event={"ID":"3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082","Type":"ContainerDied","Data":"4e44bab43dcbb59b7e726a3f51a4b666bd58495657c22c00a6323feb92348317"} Nov 01 14:48:41.521257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.521245 5473 scope.go:122] "RemoveContainer" containerID="56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0" Nov 01 14:48:41.523242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.523203 5473 generic.go:356] "Generic (PLEG): container finished" podID="f834c040-a487-468d-9806-e517879c1675" containerID="1e40cf29b994f0a638a0a21fb9d6d71485f4a970f7e7566786d466c6ca68a552" exitCode=0 Nov 01 14:48:41.523343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.523296 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" event={"ID":"f834c040-a487-468d-9806-e517879c1675","Type":"ContainerDied","Data":"1e40cf29b994f0a638a0a21fb9d6d71485f4a970f7e7566786d466c6ca68a552"} Nov 01 14:48:41.531816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.531797 5473 scope.go:122] "RemoveContainer" containerID="56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0" Nov 01 14:48:41.532114 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:48:41.532095 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0\": container with ID starting with 56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0 not found: ID does not exist" containerID="56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0" Nov 01 14:48:41.532184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.532121 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0"} err="failed to get container status \"56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0\": rpc error: code = NotFound desc = could not find container \"56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0\": container with ID starting with 56f5fca78d26ce5811678ca89e898d006c562948934675eb639eb9f7fa6ee3c0 not found: ID does not exist" Nov 01 14:48:41.688981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.688947 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl"] Nov 01 14:48:41.691957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:41.691934 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-r6vbl"] Nov 01 14:48:42.516191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.516161 5473 scope.go:122] "RemoveContainer" containerID="ab67ec7cae69d36d0f681508691022b4fda4354ede1966148d5f9595305cc918" Nov 01 14:48:42.678239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.678200 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:42.738743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.738712 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:42.836682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.836587 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr\" (UniqueName: \"kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr\") pod \"f834c040-a487-468d-9806-e517879c1675\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " Nov 01 14:48:42.836682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.836644 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca\") pod \"f834c040-a487-468d-9806-e517879c1675\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " Nov 01 14:48:42.836917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.836770 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca\") pod \"f834c040-a487-468d-9806-e517879c1675\" (UID: \"f834c040-a487-468d-9806-e517879c1675\") " Nov 01 14:48:42.837535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.837475 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca" pod "f834c040-a487-468d-9806-e517879c1675" (UID: "f834c040-a487-468d-9806-e517879c1675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:48:42.838957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.838936 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr" pod "f834c040-a487-468d-9806-e517879c1675" (UID: "f834c040-a487-468d-9806-e517879c1675"). InnerVolumeSpecName "kube-api-access-k5fpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:48:42.839182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.839153 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca" pod "f834c040-a487-468d-9806-e517879c1675" (UID: "f834c040-a487-468d-9806-e517879c1675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:48:42.938456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.938431 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f834c040-a487-468d-9806-e517879c1675-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:42.938456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.938454 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5fpr\" (UniqueName: \"kubernetes.io/projected/f834c040-a487-468d-9806-e517879c1675-kube-api-access-k5fpr\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:42.938612 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:42.938466 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f834c040-a487-468d-9806-e517879c1675-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:43.082479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.082443 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" path="/var/lib/kubelet/pods/3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082/volumes" Nov 01 14:48:43.285984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.285955 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst"] Nov 01 14:48:43.549050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.549023 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" Nov 01 14:48:43.549050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.549034 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk" event={"ID":"f834c040-a487-468d-9806-e517879c1675","Type":"ContainerDied","Data":"d7494a426e1bb13eb44b2183059b1627a587315fc9bd1d3665b425f0feb53c32"} Nov 01 14:48:43.549292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.549084 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7494a426e1bb13eb44b2183059b1627a587315fc9bd1d3665b425f0feb53c32" Nov 01 14:48:43.616995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.616946 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:43.808513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:43.808410 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:44.033157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:44.033128 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:47.787707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:47.787668 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:48.286279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.286236 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" podUID="9b4c07c9-7534-4872-808f-d5304f981dda" containerName="payload-processing" containerID="cri-o://24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d" gracePeriod=25 Nov 01 14:48:48.549030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.548973 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:48:48.587152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.587123 5473 generic.go:356] "Generic (PLEG): container finished" podID="9b4c07c9-7534-4872-808f-d5304f981dda" containerID="24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d" exitCode=0 Nov 01 14:48:48.587352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.587170 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" Nov 01 14:48:48.587352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.587210 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" event={"ID":"9b4c07c9-7534-4872-808f-d5304f981dda","Type":"ContainerDied","Data":"24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d"} Nov 01 14:48:48.587352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.587320 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst" event={"ID":"9b4c07c9-7534-4872-808f-d5304f981dda","Type":"ContainerDied","Data":"428f520df0051d7a3de92db9099ceed4c64c44be62221324812f9ae329784617"} Nov 01 14:48:48.587352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.587349 5473 scope.go:122] "RemoveContainer" containerID="24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d" Nov 01 14:48:48.597500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.597479 5473 scope.go:122] "RemoveContainer" containerID="24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d" Nov 01 14:48:48.597850 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:48:48.597819 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d\": container with ID starting with 24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d not found: ID does not exist" containerID="24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d" Nov 01 14:48:48.597937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.597860 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d"} err="failed to get container status \"24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d\": rpc error: code = NotFound desc = could not find container \"24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d\": container with ID starting with 24cff10a3a47726e54b7c4d96cc9fbb402c4ebe231f0a90c0ae6b782efd9914d not found: ID does not exist" Nov 01 14:48:48.618653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.618633 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume\") pod \"9b4c07c9-7534-4872-808f-d5304f981dda\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " Nov 01 14:48:48.618777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.618702 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz\" (UniqueName: \"kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz\") pod \"9b4c07c9-7534-4872-808f-d5304f981dda\" (UID: \"9b4c07c9-7534-4872-808f-d5304f981dda\") " Nov 01 14:48:48.619026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.619004 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume" pod "9b4c07c9-7534-4872-808f-d5304f981dda" (UID: "9b4c07c9-7534-4872-808f-d5304f981dda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:48:48.619145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.619128 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b4c07c9-7534-4872-808f-d5304f981dda-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:48.620723 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.620703 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz" pod "9b4c07c9-7534-4872-808f-d5304f981dda" (UID: "9b4c07c9-7534-4872-808f-d5304f981dda"). InnerVolumeSpecName "kube-api-access-zpmdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:48:48.720588 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:48.720546 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpmdz\" (UniqueName: \"kubernetes.io/projected/9b4c07c9-7534-4872-808f-d5304f981dda-kube-api-access-zpmdz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:48:49.009297 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:49.009265 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst"] Nov 01 14:48:49.012367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:49.012341 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-34d384-5b568b58d9-mrxst"] Nov 01 14:48:49.082404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:49.082376 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4c07c9-7534-4872-808f-d5304f981dda" path="/var/lib/kubelet/pods/9b4c07c9-7534-4872-808f-d5304f981dda/volumes" Nov 01 14:48:50.176987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:50.176951 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:52.802511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:52.802476 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:53.821050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:53.821011 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:55.978794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:55.978758 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:57.565005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:57.564965 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:58.072973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:58.072907 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:58.246874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:58.246831 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:59.323416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:59.323379 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:59.661114 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:59.661024 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:48:59.836626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:48:59.836588 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:01.134837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:01.134802 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:01.182915 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:01.180615 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:49:01.510287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:01.510201 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c"] Nov 01 14:49:01.510690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:01.510656 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" containerID="cri-o://f7a440f0933c560e31ecfc9955c16e649c0566024c24323635195209a664eda2" gracePeriod=30 Nov 01 14:49:01.550647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:01.550600 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:01.833181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:01.833144 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:02.099080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:02.098994 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr"] Nov 01 14:49:02.378751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:02.378633 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:02.552684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:02.552641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:02.930930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:02.930878 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:03.467477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:03.467445 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:03.700237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:03.699929 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:04.559343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:04.559308 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:06.546453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.546426 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Nov 01 14:49:06.546731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.546484 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Nov 01 14:49:06.778112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.778050 5473 generic.go:356] "Generic (PLEG): container finished" podID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerID="f7a440f0933c560e31ecfc9955c16e649c0566024c24323635195209a664eda2" exitCode=0 Nov 01 14:49:06.778272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.778119 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" event={"ID":"53af0abe-a1e5-49cc-919f-00324f85a09b","Type":"ContainerDied","Data":"f7a440f0933c560e31ecfc9955c16e649c0566024c24323635195209a664eda2"} Nov 01 14:49:06.778272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.778204 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" event={"ID":"53af0abe-a1e5-49cc-919f-00324f85a09b","Type":"ContainerDied","Data":"db19b9fd74df26fcfdf31d056ea0c7fc9b06db695aab467be2530c6cc39e54db"} Nov 01 14:49:06.778272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.778250 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db19b9fd74df26fcfdf31d056ea0c7fc9b06db695aab467be2530c6cc39e54db" Nov 01 14:49:06.778272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.778266 5473 scope.go:122] "RemoveContainer" containerID="77700b1e13952cda63766458fbedeb372d6c8c79741e41cd8795bb9bc68c576e" Nov 01 14:49:06.783817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.783797 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:49:06.933620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933596 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933646 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933696 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933766 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933811 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933830 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933861 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.933935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.933899 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.934160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934053 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket\") pod \"53af0abe-a1e5-49cc-919f-00324f85a09b\" (UID: \"53af0abe-a1e5-49cc-919f-00324f85a09b\") " Nov 01 14:49:06.934160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934037 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:06.934160 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934074 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:06.934380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934162 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:49:06.934380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934300 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:06.934380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934318 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:06.934633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934611 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53af0abe-a1e5-49cc-919f-00324f85a09b-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:06.934690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934640 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:06.934690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934654 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:06.934690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934667 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:06.934690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.934680 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:06.936177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.936140 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "kube-api-access-v2c9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:06.936474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.936340 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:06.936474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.936345 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:49:06.936474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:06.936364 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy" pod "53af0abe-a1e5-49cc-919f-00324f85a09b" (UID: "53af0abe-a1e5-49cc-919f-00324f85a09b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:07.035960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.035891 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:07.035960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.035920 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2c9m\" (UniqueName: \"kubernetes.io/projected/53af0abe-a1e5-49cc-919f-00324f85a09b-kube-api-access-v2c9m\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:07.035960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.035930 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:07.035960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.035939 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53af0abe-a1e5-49cc-919f-00324f85a09b-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:07.099860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.099826 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" podUID="7d5cd57e-fb93-498e-8c71-90e54883e644" containerName="payload-pre-processing" containerID="cri-o://aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432" gracePeriod=25 Nov 01 14:49:07.339782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.339758 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:49:07.440113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.440086 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv\" (UniqueName: \"kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv\") pod \"7d5cd57e-fb93-498e-8c71-90e54883e644\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " Nov 01 14:49:07.440282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.440231 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume\") pod \"7d5cd57e-fb93-498e-8c71-90e54883e644\" (UID: \"7d5cd57e-fb93-498e-8c71-90e54883e644\") " Nov 01 14:49:07.440582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.440548 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume" pod "7d5cd57e-fb93-498e-8c71-90e54883e644" (UID: "7d5cd57e-fb93-498e-8c71-90e54883e644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:49:07.440741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.440727 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d5cd57e-fb93-498e-8c71-90e54883e644-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:07.442203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.442180 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv" pod "7d5cd57e-fb93-498e-8c71-90e54883e644" (UID: "7d5cd57e-fb93-498e-8c71-90e54883e644"). InnerVolumeSpecName "kube-api-access-hxqsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:07.542225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.542190 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqsv\" (UniqueName: \"kubernetes.io/projected/7d5cd57e-fb93-498e-8c71-90e54883e644-kube-api-access-hxqsv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:07.787965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.787940 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c" Nov 01 14:49:07.789620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.789589 5473 generic.go:356] "Generic (PLEG): container finished" podID="7d5cd57e-fb93-498e-8c71-90e54883e644" containerID="aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432" exitCode=0 Nov 01 14:49:07.789753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.789638 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" Nov 01 14:49:07.789753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.789678 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" event={"ID":"7d5cd57e-fb93-498e-8c71-90e54883e644","Type":"ContainerDied","Data":"aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432"} Nov 01 14:49:07.789753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.789741 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr" event={"ID":"7d5cd57e-fb93-498e-8c71-90e54883e644","Type":"ContainerDied","Data":"9b7328588d7e4ff3eea0d8f658c9ce37dc38f415394f441e2cc19cd02ba6f36b"} Nov 01 14:49:07.789926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.789770 5473 scope.go:122] "RemoveContainer" containerID="aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432" Nov 01 14:49:07.803296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.803277 5473 scope.go:122] "RemoveContainer" containerID="aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432" Nov 01 14:49:07.803572 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:07.803550 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432\": container with ID starting with aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432 not found: ID does not exist" containerID="aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432" Nov 01 14:49:07.803625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.803582 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432"} err="failed to get container status \"aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432\": rpc error: code = NotFound desc = could not find container \"aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432\": container with ID starting with aedf3b2246154ce2ac69e8b24e1b1322cd8c5cf463bd684145e0032c313fe432 not found: ID does not exist" Nov 01 14:49:07.930678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:07.930641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:08.019596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:08.019567 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr"] Nov 01 14:49:08.022402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:08.022382 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c335a-6dd647b6f9-8fbzr"] Nov 01 14:49:08.132085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:08.132002 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c"] Nov 01 14:49:08.134563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:08.134533 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:08.135406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:08.135382 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-cbac83-openshift-default-cbc65c6b9-g9l9c"] Nov 01 14:49:08.479532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:08.479452 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:09.081512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:09.081484 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" path="/var/lib/kubelet/pods/53af0abe-a1e5-49cc-919f-00324f85a09b/volumes" Nov 01 14:49:09.081909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:09.081895 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d5cd57e-fb93-498e-8c71-90e54883e644" path="/var/lib/kubelet/pods/7d5cd57e-fb93-498e-8c71-90e54883e644/volumes" Nov 01 14:49:10.986409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:10.986373 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:11.158962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:11.158915 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:11.319973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:11.319940 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:15.419511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:15.419471 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:19.886608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:19.886574 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:24.402998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:24.402926 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:26.620971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:26.620903 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:26.809618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:26.809591 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:29.117835 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:29.117794 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:30.696308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:30.696257 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:31.106721 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:31.106688 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:49:33.936376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:33.936335 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:34.384223 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:34.384176 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:35.198668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:35.198639 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:36.415111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:36.415080 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv"] Nov 01 14:49:36.415571 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:36.415294 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" containerID="cri-o://88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979" gracePeriod=30 Nov 01 14:49:36.466751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:36.466712 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz"] Nov 01 14:49:36.466987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:36.466956 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" containerID="cri-o://00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5" gracePeriod=30 Nov 01 14:49:36.978593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:36.978566 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:41.492411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.492377 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:41.700999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.700973 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:49:41.742016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.741988 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742037 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742075 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742137 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742168 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742254 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742293 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.742846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742422 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742431 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.742846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742555 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.742846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742595 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.742846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742643 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket\") pod \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\" (UID: \"17fde8df-a3c5-470e-9e6d-0df9fa6d6576\") " Nov 01 14:49:41.743174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742948 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.743174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.742955 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:49:41.743340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.743263 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.743340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.743283 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.743340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.743299 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.743340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.743313 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.743340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.743327 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.744607 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.744584 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "kube-api-access-6sjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:41.744817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.744778 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:49:41.744923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.744879 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:41.744968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.744925 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy" pod "17fde8df-a3c5-470e-9e6d-0df9fa6d6576" (UID: "17fde8df-a3c5-470e-9e6d-0df9fa6d6576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.750964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.750945 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:49:41.844427 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844403 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844435 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844471 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844534 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844579 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844616 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844679 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844729 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844833 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844753 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:49:41.844901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844835 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs\") pod \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\" (UID: \"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6\") " Nov 01 14:49:41.844901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.844851 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.845038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845016 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.845121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845049 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.845121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845085 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.845452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845426 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845456 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845473 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845487 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845502 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845518 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845532 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845547 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.845576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.845562 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjwf\" (UniqueName: \"kubernetes.io/projected/17fde8df-a3c5-470e-9e6d-0df9fa6d6576-kube-api-access-6sjwf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.846773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.846748 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "kube-api-access-qpb6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:41.846872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.846768 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:49:41.846961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.846940 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:41.846961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.846954 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token" pod "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" (UID: "20b71d38-9b23-45a7-8eeb-6a14f3adb9b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:41.946976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.946898 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.946976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.946933 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.946976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.946949 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:41.946976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:41.946964 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpb6j\" (UniqueName: \"kubernetes.io/projected/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6-kube-api-access-qpb6j\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:42.028179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.028150 5473 generic.go:356] "Generic (PLEG): container finished" podID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerID="88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979" exitCode=0 Nov 01 14:49:42.028382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.028195 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" Nov 01 14:49:42.028382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.028244 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" event={"ID":"17fde8df-a3c5-470e-9e6d-0df9fa6d6576","Type":"ContainerDied","Data":"88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979"} Nov 01 14:49:42.028382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.028305 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" event={"ID":"17fde8df-a3c5-470e-9e6d-0df9fa6d6576","Type":"ContainerDied","Data":"60d71a75fa667082faa951d8047c58d72b55342c887184b9bb10c2f8420023d9"} Nov 01 14:49:42.028382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.028319 5473 scope.go:122] "RemoveContainer" containerID="88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979" Nov 01 14:49:42.029819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.029799 5473 generic.go:356] "Generic (PLEG): container finished" podID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerID="00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5" exitCode=0 Nov 01 14:49:42.029925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.029821 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" event={"ID":"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6","Type":"ContainerDied","Data":"00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5"} Nov 01 14:49:42.029925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.029839 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" Nov 01 14:49:42.029925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.029871 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz" event={"ID":"20b71d38-9b23-45a7-8eeb-6a14f3adb9b6","Type":"ContainerDied","Data":"8665d1c2c84a25c66798c1e32c160c67637f985f44608daa3070765afc098fb0"} Nov 01 14:49:42.041659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.041642 5473 scope.go:122] "RemoveContainer" containerID="88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979" Nov 01 14:49:42.045639 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:42.042904 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979\": container with ID starting with 88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979 not found: ID does not exist" containerID="88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979" Nov 01 14:49:42.045639 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.042964 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979"} err="failed to get container status \"88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979\": rpc error: code = NotFound desc = could not find container \"88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979\": container with ID starting with 88979fd1dc0af0425692f267ca6f789fc91b5bdf9132a5cb3d60672e4c5e0979 not found: ID does not exist" Nov 01 14:49:42.045639 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.042984 5473 scope.go:122] "RemoveContainer" containerID="00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5" Nov 01 14:49:42.053936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.053919 5473 scope.go:122] "RemoveContainer" containerID="00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5" Nov 01 14:49:42.054190 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:42.054174 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5\": container with ID starting with 00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5 not found: ID does not exist" containerID="00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5" Nov 01 14:49:42.054287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.054194 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5"} err="failed to get container status \"00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5\": rpc error: code = NotFound desc = could not find container \"00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5\": container with ID starting with 00fbe9e2724019eb1902dbfeb7958a35ab4288bfd122d461c55d3b1c3951d1f5 not found: ID does not exist" Nov 01 14:49:42.194897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.194870 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv"] Nov 01 14:49:42.198576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.198496 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv"] Nov 01 14:49:42.306822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.306795 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz"] Nov 01 14:49:42.312356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.312334 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-8lzxz"] Nov 01 14:49:42.560940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.560913 5473 scope.go:122] "RemoveContainer" containerID="c07bf4287c07cfb525f3e7bde64a37aba868ad5c63670a95a594407c81b22034" Nov 01 14:49:42.685786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.685756 5473 patch_prober.go:29] interesting pod/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:49:42.685897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:42.685813 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-fssqv" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:49:43.081002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:43.080970 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" path="/var/lib/kubelet/pods/17fde8df-a3c5-470e-9e6d-0df9fa6d6576/volumes" Nov 01 14:49:43.081452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:43.081438 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" path="/var/lib/kubelet/pods/20b71d38-9b23-45a7-8eeb-6a14f3adb9b6/volumes" Nov 01 14:49:45.396511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:45.396470 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:45.841789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:45.841752 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:45.997743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:45.997714 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:47.251374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:47.251343 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:49.526271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:49.526202 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:49.767783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:49.767745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:50.589440 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:50.589405 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:50.856977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:50.856894 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:51.163051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:51.162972 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:51.296387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:51.296353 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj"] Nov 01 14:49:51.296635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:51.296611 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" containerID="cri-o://73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec" gracePeriod=30 Nov 01 14:49:51.325163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:51.325130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:52.017823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.017787 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:49:52.018308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.017861 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:49:52.077034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.076999 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:52.323303 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323199 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm"] Nov 01 14:49:52.323794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323779 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9b4c07c9-7534-4872-808f-d5304f981dda" containerName="payload-processing" Nov 01 14:49:52.323837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323796 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4c07c9-7534-4872-808f-d5304f981dda" containerName="payload-processing" Nov 01 14:49:52.323837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323808 5473 cpu_manager.go:414] "containerMap: removing container" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" Nov 01 14:49:52.323837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323815 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" Nov 01 14:49:52.323837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323823 5473 cpu_manager.go:414] "containerMap: removing container" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" Nov 01 14:49:52.323837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323829 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323853 5473 cpu_manager.go:414] "containerMap: removing container" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323860 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323880 5473 cpu_manager.go:414] "containerMap: removing container" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323884 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323895 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7d5cd57e-fb93-498e-8c71-90e54883e644" containerName="payload-pre-processing" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323899 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d5cd57e-fb93-498e-8c71-90e54883e644" containerName="payload-pre-processing" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323908 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f834c040-a487-468d-9806-e517879c1675" containerName="revoke-keys" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323912 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f834c040-a487-468d-9806-e517879c1675" containerName="revoke-keys" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323920 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" Nov 01 14:49:52.323997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.323925 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324009 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324037 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fde8df-a3c5-470e-9e6d-0df9fa6d6576" containerName="istio-proxy" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324045 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d5cd57e-fb93-498e-8c71-90e54883e644" containerName="payload-pre-processing" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324052 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f834c040-a487-468d-9806-e517879c1675" containerName="revoke-keys" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324057 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbbd1b0-ba54-4cb3-ac66-58dbee7bf082" containerName="istio-proxy" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324066 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4c07c9-7534-4872-808f-d5304f981dda" containerName="payload-processing" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324075 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="20b71d38-9b23-45a7-8eeb-6a14f3adb9b6" containerName="istio-proxy" Nov 01 14:49:52.324357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.324081 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="53af0abe-a1e5-49cc-919f-00324f85a09b" containerName="istio-proxy" Nov 01 14:49:52.329050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.329032 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.384466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.384434 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm"] Nov 01 14:49:52.430232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.430184 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-60547a-openshift-default-dockercfg-w2ww9\"" Nov 01 14:49:52.435651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.435625 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:52.481925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.481892 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.481932 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482016 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482049 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482097 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482166 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482209 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482264 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.482367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.482313 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2b2q\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.583973 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584039 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584086 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2b2q\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584148 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584263 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584305 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584337 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584396 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584466 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584707 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.584839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.584794 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.585054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.585026 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.585290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.585257 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.585360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.585324 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.586641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.586617 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.587024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.587005 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.642308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.642269 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.642465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.642380 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2b2q\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q\") pod \"e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:52.766172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.766139 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:52.942540 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:52.942465 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:53.151862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.151827 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:53.320465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.320433 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:53.359528 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:49:53.359502 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cbaae7_fb6f_468c_85a3_5d377df8e971.slice/crio-79974231676a3cde85fa874ff0472f0e1127a6e1f479881b32e050827db0c007 WatchSource:0}: Error finding container 79974231676a3cde85fa874ff0472f0e1127a6e1f479881b32e050827db0c007: Status 404 returned error can't find the container with id 79974231676a3cde85fa874ff0472f0e1127a6e1f479881b32e050827db0c007 Nov 01 14:49:53.362012 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.361980 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:49:53.362106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.362057 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:49:53.362106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.362091 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:49:53.362677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.362656 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm"] Nov 01 14:49:53.820630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:53.820593 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:54.107684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:54.107595 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" event={"ID":"c8cbaae7-fb6f-468c-85a3-5d377df8e971","Type":"ContainerStarted","Data":"c88fea85342ef19a2f5070940b366bb3dbe708a17bf40aa775f757d2622bbda5"} Nov 01 14:49:54.107684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:54.107650 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" event={"ID":"c8cbaae7-fb6f-468c-85a3-5d377df8e971","Type":"ContainerStarted","Data":"79974231676a3cde85fa874ff0472f0e1127a6e1f479881b32e050827db0c007"} Nov 01 14:49:54.279073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:54.279021 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podStartSLOduration=2.279006663 podStartE2EDuration="2.279006663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:49:54.275483017 +0000 UTC m=+1933.333634865" watchObservedRunningTime="2026-11-01 14:49:54.279006663 +0000 UTC m=+1933.337158511" Nov 01 14:49:54.943436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:54.943406 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:54.948854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:54.948832 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:55.114912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:55.114885 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:55.115904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:55.115881 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:49:56.555771 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.555748 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:49:56.736444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736356 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736417 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736472 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736551 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736621 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736659 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736685 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736734 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.736979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.736808 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs\") pod \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\" (UID: \"bc47b33b-ef0e-44d5-9097-d4fa413682f5\") " Nov 01 14:49:56.737255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.737201 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:56.737619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.737594 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:56.737619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.737324 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:56.737779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.737690 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:56.737897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.737793 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:49:56.738056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738038 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.738127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738061 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.738127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738075 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.738127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738088 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.738127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738101 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.738953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738931 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:49:56.739038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.738928 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:49:56.739099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.739049 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:56.739499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.739474 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc" pod "bc47b33b-ef0e-44d5-9097-d4fa413682f5" (UID: "bc47b33b-ef0e-44d5-9097-d4fa413682f5"). InnerVolumeSpecName "kube-api-access-9nwzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:49:56.839040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.839004 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwzc\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-kube-api-access-9nwzc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.839040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.839036 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.839040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.839045 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.839296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.839054 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc47b33b-ef0e-44d5-9097-d4fa413682f5-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:49:56.947203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:56.947164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:57.129962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.129933 5473 generic.go:356] "Generic (PLEG): container finished" podID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerID="73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec" exitCode=0 Nov 01 14:49:57.130154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.129981 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" Nov 01 14:49:57.130154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.130017 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" event={"ID":"bc47b33b-ef0e-44d5-9097-d4fa413682f5","Type":"ContainerDied","Data":"73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec"} Nov 01 14:49:57.130154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.130079 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj" event={"ID":"bc47b33b-ef0e-44d5-9097-d4fa413682f5","Type":"ContainerDied","Data":"e60ec6503bdab91a97c087dccbfba6734680f04fb413c8cbde8ba4a7ecaaecdc"} Nov 01 14:49:57.130154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.130108 5473 scope.go:122] "RemoveContainer" containerID="73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec" Nov 01 14:49:57.140504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.140487 5473 scope.go:122] "RemoveContainer" containerID="ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0" Nov 01 14:49:57.150194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.150176 5473 scope.go:122] "RemoveContainer" containerID="73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec" Nov 01 14:49:57.150530 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:57.150501 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec\": container with ID starting with 73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec not found: ID does not exist" containerID="73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec" Nov 01 14:49:57.150623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.150540 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec"} err="failed to get container status \"73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec\": rpc error: code = NotFound desc = could not find container \"73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec\": container with ID starting with 73b4c9e08fa17060f8d497eaac397f0255c033c17f373a686b9f74e2dcdee5ec not found: ID does not exist" Nov 01 14:49:57.150623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.150590 5473 scope.go:122] "RemoveContainer" containerID="ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0" Nov 01 14:49:57.150856 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:49:57.150834 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0\": container with ID starting with ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0 not found: ID does not exist" containerID="ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0" Nov 01 14:49:57.150922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.150860 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0"} err="failed to get container status \"ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0\": rpc error: code = NotFound desc = could not find container \"ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0\": container with ID starting with ca85812e2a1583bad9d875d98926cb158deccf1d404faba4d74b7814edc580b0 not found: ID does not exist" Nov 01 14:49:57.151001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.150988 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:57.307674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.307647 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj"] Nov 01 14:49:57.321439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.321407 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-25ce0b-openshift-default-646c4d884f-6p5nj"] Nov 01 14:49:57.491292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:57.491201 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:58.688376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:58.688339 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:58.867586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:58.867556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:49:59.082127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:49:59.082096 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" path="/var/lib/kubelet/pods/bc47b33b-ef0e-44d5-9097-d4fa413682f5/volumes" Nov 01 14:50:00.053971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.053941 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:00.264801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.264763 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:00.424544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.424470 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh"] Nov 01 14:50:00.424983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.424970 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" Nov 01 14:50:00.425031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.424984 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" Nov 01 14:50:00.425102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.425094 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" Nov 01 14:50:00.429946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.429928 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.486581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.486550 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh"] Nov 01 14:50:00.493338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.492594 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqbvw\" (UniqueName: \"kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw\") pod \"payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.493338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.492710 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.537297 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.536419 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-60547a-dockercfg-gcsv8\"" Nov 01 14:50:00.537297 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.536501 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-60547a\"" Nov 01 14:50:00.592290 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.592259 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg"] Nov 01 14:50:00.593034 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.593019 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" Nov 01 14:50:00.593087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.593037 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" Nov 01 14:50:00.593190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.593180 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc47b33b-ef0e-44d5-9097-d4fa413682f5" containerName="istio-proxy" Nov 01 14:50:00.594035 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.594015 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqbvw\" (UniqueName: \"kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw\") pod \"payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.594185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.594100 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.594730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.594713 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.597075 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.597057 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.647512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.647483 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg"] Nov 01 14:50:00.651828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.651798 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqbvw\" (UniqueName: \"kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw\") pod \"payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.695031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.694958 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.695145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.695102 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2bwh\" (UniqueName: \"kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh\") pod \"payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.742955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.742929 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:00.796599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.796447 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2bwh\" (UniqueName: \"kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh\") pod \"payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.796697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.796637 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.798241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.797527 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.853974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.853941 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2bwh\" (UniqueName: \"kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh\") pod \"payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:00.910644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:00.910619 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:01.107411 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:50:01.107376 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:50:01.132561 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:50:01.132526 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243c1e36_16c7_428f_a8be_09be0e50eb60.slice/crio-cc0f0f777b7d4ff78055c5a29a745be305c2f3410e7bfd5156fb341e8ba82e28 WatchSource:0}: Error finding container cc0f0f777b7d4ff78055c5a29a745be305c2f3410e7bfd5156fb341e8ba82e28: Status 404 returned error can't find the container with id cc0f0f777b7d4ff78055c5a29a745be305c2f3410e7bfd5156fb341e8ba82e28 Nov 01 14:50:01.138262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:01.137578 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh"] Nov 01 14:50:01.199692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:01.199666 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" event={"ID":"243c1e36-16c7-428f-a8be-09be0e50eb60","Type":"ContainerStarted","Data":"cc0f0f777b7d4ff78055c5a29a745be305c2f3410e7bfd5156fb341e8ba82e28"} Nov 01 14:50:01.277274 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:50:01.277249 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd2564b_117c_44c8_aa28_152ec30ef7b5.slice/crio-b92688fcb3e9f6de66d691062bd250a42b387cadfbfcad06408da9801ca2dc98 WatchSource:0}: Error finding container b92688fcb3e9f6de66d691062bd250a42b387cadfbfcad06408da9801ca2dc98: Status 404 returned error can't find the container with id b92688fcb3e9f6de66d691062bd250a42b387cadfbfcad06408da9801ca2dc98 Nov 01 14:50:01.280309 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:01.280286 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg"] Nov 01 14:50:02.208202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.208166 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" event={"ID":"243c1e36-16c7-428f-a8be-09be0e50eb60","Type":"ContainerStarted","Data":"098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c"} Nov 01 14:50:02.208714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.208247 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:02.209815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.209789 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" event={"ID":"0dd2564b-117c-44c8-aa28-152ec30ef7b5","Type":"ContainerStarted","Data":"660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6"} Nov 01 14:50:02.209948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.209831 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" event={"ID":"0dd2564b-117c-44c8-aa28-152ec30ef7b5","Type":"ContainerStarted","Data":"b92688fcb3e9f6de66d691062bd250a42b387cadfbfcad06408da9801ca2dc98"} Nov 01 14:50:02.209948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.209904 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:02.322489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.322443 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" podStartSLOduration=2.322430174 podStartE2EDuration="2.322430174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:02.321006406 +0000 UTC m=+1941.379158256" watchObservedRunningTime="2026-11-01 14:50:02.322430174 +0000 UTC m=+1941.380582019" Nov 01 14:50:02.440371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:02.440308 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" podStartSLOduration=2.440289301 podStartE2EDuration="2.440289301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:02.439363023 +0000 UTC m=+1941.497514873" watchObservedRunningTime="2026-11-01 14:50:02.440289301 +0000 UTC m=+1941.498441152" Nov 01 14:50:05.098633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:05.098600 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:05.269327 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:05.269293 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:05.453167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:05.453092 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:06.971681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:06.971648 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:07.567334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:07.567303 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:07.736769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:07.736736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:09.720316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:09.714692 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:10.360249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:10.360203 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:13.218674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:13.218645 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:50:13.219108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:13.218708 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:50:14.212244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:14.212186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:16.126474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:16.126441 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:18.846701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:18.846666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:21.115625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:21.115587 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:21.889116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:21.889083 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28"] Nov 01 14:50:21.892736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:21.892718 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:21.945303 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:21.945270 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2048c1-openshift-default-dockercfg-m944b\"" Nov 01 14:50:21.998494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:21.998465 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28"] Nov 01 14:50:22.011431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.011395 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:22.042742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.042708 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043008 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.042981 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043206 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043363 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zsv\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043400 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043451 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043500 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043536 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.043568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.043566 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.144788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.144706 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.144788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.144754 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.144788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.144776 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.144899 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145125 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145167 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145257 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145259 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145326 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zsv\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145333 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145362 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145505 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145682 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.145808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.145687 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.147061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.147043 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.147447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.147427 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.202054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.202029 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.202198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.202180 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zsv\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv\") pod \"e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.206992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.206970 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:22.311877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.311448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:22.640627 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:50:22.640592 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19b6ee9_20b0_4318_92ab_823cd52cb7dc.slice/crio-a3da784f10f1263d05498b165f09cb414681995ce195d84525ad308d7f7b2096 WatchSource:0}: Error finding container a3da784f10f1263d05498b165f09cb414681995ce195d84525ad308d7f7b2096: Status 404 returned error can't find the container with id a3da784f10f1263d05498b165f09cb414681995ce195d84525ad308d7f7b2096 Nov 01 14:50:22.643350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.643316 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:50:22.643448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.643382 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:50:22.643448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.643408 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:50:22.644481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:22.644459 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28"] Nov 01 14:50:23.079050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:23.079013 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:23.358901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:23.358804 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" event={"ID":"f19b6ee9-20b0-4318-92ab-823cd52cb7dc","Type":"ContainerStarted","Data":"8c33ec8c4a63ed0bd11389d709d02dab94294cfd761c14e395dc8a58a504e036"} Nov 01 14:50:23.358901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:23.358870 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" event={"ID":"f19b6ee9-20b0-4318-92ab-823cd52cb7dc","Type":"ContainerStarted","Data":"a3da784f10f1263d05498b165f09cb414681995ce195d84525ad308d7f7b2096"} Nov 01 14:50:23.477639 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:23.477590 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podStartSLOduration=2.47757712 podStartE2EDuration="2.47757712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:23.47591641 +0000 UTC m=+1962.534068258" watchObservedRunningTime="2026-11-01 14:50:23.47757712 +0000 UTC m=+1962.535728967" Nov 01 14:50:23.581665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:23.581631 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:23.738801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:23.738725 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:24.207391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:24.207360 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:24.212520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:24.212494 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:24.365676 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:24.365647 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:24.366469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:24.366452 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:50:25.025137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:25.025108 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:26.569360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:26.569332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:26.888757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:26.888683 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:29.668518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:29.668482 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:29.827456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:29.827419 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:30.186407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.186372 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7"] Nov 01 14:50:30.191663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.191640 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.242575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.242544 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7"] Nov 01 14:50:30.246061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.246027 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.246345 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.246325 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkh7h\" (UniqueName: \"kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h\") pod \"payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.288433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.288404 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2048c1\"" Nov 01 14:50:30.288821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.288804 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2048c1-dockercfg-w88k5\"" Nov 01 14:50:30.347899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.347853 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.347899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.347903 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkh7h\" (UniqueName: \"kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h\") pod \"payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.349177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.349150 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.405907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.405875 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkh7h\" (UniqueName: \"kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h\") pod \"payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.507442 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.507373 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:30.873573 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:50:30.873543 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cd6e5c_7a1d_4a8b_a89e_a24d44072ef1.slice/crio-9a3f4a92cae97b52b930a1cc0fc2733ee9bcf11345e41ab48da59a2fb8654057 WatchSource:0}: Error finding container 9a3f4a92cae97b52b930a1cc0fc2733ee9bcf11345e41ab48da59a2fb8654057: Status 404 returned error can't find the container with id 9a3f4a92cae97b52b930a1cc0fc2733ee9bcf11345e41ab48da59a2fb8654057 Nov 01 14:50:30.877059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:30.877032 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7"] Nov 01 14:50:31.107149 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:50:31.107116 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:50:31.418981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:31.418934 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:31.422653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:31.422623 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" event={"ID":"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1","Type":"ContainerStarted","Data":"b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230"} Nov 01 14:50:31.422772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:31.422681 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" event={"ID":"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1","Type":"ContainerStarted","Data":"9a3f4a92cae97b52b930a1cc0fc2733ee9bcf11345e41ab48da59a2fb8654057"} Nov 01 14:50:31.422822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:31.422800 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:31.539376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:31.539317 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" podStartSLOduration=1.5393014950000001 podStartE2EDuration="1.539301495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:31.537482544 +0000 UTC m=+1970.595634391" watchObservedRunningTime="2026-11-01 14:50:31.539301495 +0000 UTC m=+1970.597453343" Nov 01 14:50:31.587207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:31.587168 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:32.964225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:32.964177 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:33.223002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.222928 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744"] Nov 01 14:50:33.223148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.223110 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" containerID="cri-o://7c225c26503d86f7f8a10f48053b116cce54b16c404d3ff74c904be4bf7e4c9e" gracePeriod=30 Nov 01 14:50:33.584565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.584531 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855"] Nov 01 14:50:33.584765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.584728 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" containerID="cri-o://1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830" gracePeriod=30 Nov 01 14:50:33.681884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.681840 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:33.714888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.714852 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:50:33.715049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:33.714912 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:50:34.258392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:34.258359 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:36.458330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:36.458298 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:38.232998 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.232960 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": EOF" start-of-body= Nov 01 14:50:38.233433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.233024 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": EOF" Nov 01 14:50:38.473617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.473588 5473 generic.go:356] "Generic (PLEG): container finished" podID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerID="7c225c26503d86f7f8a10f48053b116cce54b16c404d3ff74c904be4bf7e4c9e" exitCode=0 Nov 01 14:50:38.473764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.473660 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" event={"ID":"5d6bf672-93b8-4f7c-8531-e658951426dc","Type":"ContainerDied","Data":"7c225c26503d86f7f8a10f48053b116cce54b16c404d3ff74c904be4bf7e4c9e"} Nov 01 14:50:38.499274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.499223 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:50:38.541275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541211 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541335 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541370 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541418 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541481 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.541513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541501 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541567 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541618 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541666 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541737 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert\") pod \"5d6bf672-93b8-4f7c-8531-e658951426dc\" (UID: \"5d6bf672-93b8-4f7c-8531-e658951426dc\") " Nov 01 14:50:38.541834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541747 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.541946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541888 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.541946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.541901 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.542256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.542233 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:50:38.542373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.542300 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.542373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.542319 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.542373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.542332 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.542373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.542341 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.543628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.543602 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.543734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.543716 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:50:38.543926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.543906 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:50:38.543926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.543915 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn" pod "5d6bf672-93b8-4f7c-8531-e658951426dc" (UID: "5d6bf672-93b8-4f7c-8531-e658951426dc"). InnerVolumeSpecName "kube-api-access-ftdsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:50:38.643897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.643869 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.643897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.643894 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.644054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.643904 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdsn\" (UniqueName: \"kubernetes.io/projected/5d6bf672-93b8-4f7c-8531-e658951426dc-kube-api-access-ftdsn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.644054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.643913 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6bf672-93b8-4f7c-8531-e658951426dc-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.644054 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.643923 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6bf672-93b8-4f7c-8531-e658951426dc-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.838475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.838451 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:50:38.947307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947274 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947332 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947401 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947460 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947484 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947533 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947636 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.947790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947645 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947698 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947766 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy\") pod \"b442603a-e1bb-460f-aa23-7a0de15c2f84\" (UID: \"b442603a-e1bb-460f-aa23-7a0de15c2f84\") " Nov 01 14:50:38.947790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947779 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.948079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.947866 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:50:38.948079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948027 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.948201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948077 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.948353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948330 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.948421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948359 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b442603a-e1bb-460f-aa23-7a0de15c2f84-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.948421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948376 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.948421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948390 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.948421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.948404 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:38.949659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.949636 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:50:38.949734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.949712 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:38.949828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.949813 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:50:38.949889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:38.949876 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv" pod "b442603a-e1bb-460f-aa23-7a0de15c2f84" (UID: "b442603a-e1bb-460f-aa23-7a0de15c2f84"). InnerVolumeSpecName "kube-api-access-xsncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:50:39.049448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.049416 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:39.049448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.049443 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsncv\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-kube-api-access-xsncv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:39.049630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.049459 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:39.049630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.049471 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b442603a-e1bb-460f-aa23-7a0de15c2f84-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:39.482558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.482471 5473 generic.go:356] "Generic (PLEG): container finished" podID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerID="1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830" exitCode=0 Nov 01 14:50:39.482558 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.482527 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" Nov 01 14:50:39.483059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.482555 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" event={"ID":"b442603a-e1bb-460f-aa23-7a0de15c2f84","Type":"ContainerDied","Data":"1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830"} Nov 01 14:50:39.483059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.482621 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855" event={"ID":"b442603a-e1bb-460f-aa23-7a0de15c2f84","Type":"ContainerDied","Data":"da6e1871ba925f6e0b59815f60c3256929476a2950fcb0522973d4f48980c828"} Nov 01 14:50:39.483059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.482654 5473 scope.go:122] "RemoveContainer" containerID="1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830" Nov 01 14:50:39.484451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.484427 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" Nov 01 14:50:39.484451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.484436 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744" event={"ID":"5d6bf672-93b8-4f7c-8531-e658951426dc","Type":"ContainerDied","Data":"82316e327c6e36ddf6d613887c25a80a7ccb6e6f8f3190746f5526a3e3e258ee"} Nov 01 14:50:39.493159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.493142 5473 scope.go:122] "RemoveContainer" containerID="99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad" Nov 01 14:50:39.503103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.503083 5473 scope.go:122] "RemoveContainer" containerID="1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830" Nov 01 14:50:39.503434 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:50:39.503411 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830\": container with ID starting with 1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830 not found: ID does not exist" containerID="1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830" Nov 01 14:50:39.503515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.503445 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830"} err="failed to get container status \"1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830\": rpc error: code = NotFound desc = could not find container \"1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830\": container with ID starting with 1b3142da6c2b532ba8bcb937e1d466dbe25dd9fe0d2b617677ca8d05f09b2830 not found: ID does not exist" Nov 01 14:50:39.503515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.503467 5473 scope.go:122] "RemoveContainer" containerID="99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad" Nov 01 14:50:39.503708 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:50:39.503690 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad\": container with ID starting with 99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad not found: ID does not exist" containerID="99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad" Nov 01 14:50:39.503765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.503713 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad"} err="failed to get container status \"99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad\": rpc error: code = NotFound desc = could not find container \"99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad\": container with ID starting with 99ead3e933afb68867aa500bc5fba8ca98115b40d4da191adb013aab58c662ad not found: ID does not exist" Nov 01 14:50:39.503765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.503729 5473 scope.go:122] "RemoveContainer" containerID="7c225c26503d86f7f8a10f48053b116cce54b16c404d3ff74c904be4bf7e4c9e" Nov 01 14:50:39.650703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.650667 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744"] Nov 01 14:50:39.700553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.700526 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-r9744"] Nov 01 14:50:39.810834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.810806 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855"] Nov 01 14:50:39.815429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:39.815404 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-71e39a-openshift-default-86b7fbcfff-b9855"] Nov 01 14:50:40.332178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:40.332121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:40.649486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:40.649415 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:41.082887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:41.082856 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" path="/var/lib/kubelet/pods/5d6bf672-93b8-4f7c-8531-e658951426dc/volumes" Nov 01 14:50:41.083473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:41.083455 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" path="/var/lib/kubelet/pods/b442603a-e1bb-460f-aa23-7a0de15c2f84/volumes" Nov 01 14:50:41.194466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:41.194434 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:41.353774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:41.353701 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:42.435277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:42.435248 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:50:42.525758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:42.525727 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:46.071770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:46.071735 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:46.244130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:46.244098 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:47.421733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:47.421699 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:47.617829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:47.617795 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:47.783304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:47.783269 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:48.232046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:48.231959 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z"] Nov 01 14:50:48.232264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:48.232190 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" containerID="cri-o://69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc" gracePeriod=30 Nov 01 14:50:48.327458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:48.327423 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:50.835859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:50.835766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:51.297529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.297496 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm"] Nov 01 14:50:51.298233 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298195 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" Nov 01 14:50:51.298357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298236 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" Nov 01 14:50:51.298357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298261 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" Nov 01 14:50:51.298357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298268 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" Nov 01 14:50:51.298357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298291 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" Nov 01 14:50:51.298357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298300 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" Nov 01 14:50:51.298538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298419 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" Nov 01 14:50:51.298538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298431 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6bf672-93b8-4f7c-8531-e658951426dc" containerName="istio-proxy" Nov 01 14:50:51.298538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.298442 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b442603a-e1bb-460f-aa23-7a0de15c2f84" containerName="istio-proxy" Nov 01 14:50:51.303507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.303488 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.358667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.358631 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm"] Nov 01 14:50:51.404170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.404143 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0bb4b4-openshift-default-dockercfg-v4qg6\"" Nov 01 14:50:51.407951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.407925 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.407969 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408118 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408145 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408270 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408330 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408402 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408481 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.408539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.408516 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrlz\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.416459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.416430 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.510022 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.510074 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.510168 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.510195 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511359 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511453 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511503 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511578 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511627 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511638 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511672 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrlz\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.511880 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.512101 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.513249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.512385 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.522351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.522274 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.530598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.526040 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.569611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.569526 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrlz\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.569839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.569660 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token\") pod \"e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:51.599646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.599613 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:51.618059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:51.618034 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:52.042327 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.042300 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm"] Nov 01 14:50:52.044576 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:50:52.044546 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c12e9c_f109_4dbb_8ba2_70a7b978b7ce.slice/crio-1fc2e5feb0bc7f6b07dea85aa5976ee96c9689d4f2885967cfae4c5c0f90f5b7 WatchSource:0}: Error finding container 1fc2e5feb0bc7f6b07dea85aa5976ee96c9689d4f2885967cfae4c5c0f90f5b7: Status 404 returned error can't find the container with id 1fc2e5feb0bc7f6b07dea85aa5976ee96c9689d4f2885967cfae4c5c0f90f5b7 Nov 01 14:50:52.047201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.047163 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:50:52.047393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.047274 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:50:52.047393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.047324 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:50:52.063873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.063821 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:52.437664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.437590 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:52.492232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.492187 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:50:52.492413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.492274 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:50:52.593310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.593267 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" event={"ID":"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce","Type":"ContainerStarted","Data":"a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2"} Nov 01 14:50:52.593476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.593332 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" event={"ID":"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce","Type":"ContainerStarted","Data":"1fc2e5feb0bc7f6b07dea85aa5976ee96c9689d4f2885967cfae4c5c0f90f5b7"} Nov 01 14:50:52.603961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.603934 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:52.618984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.618964 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:52.710413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:52.710313 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podStartSLOduration=1.71029878 podStartE2EDuration="1.71029878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:50:52.709034976 +0000 UTC m=+1991.767186840" watchObservedRunningTime="2026-11-01 14:50:52.71029878 +0000 UTC m=+1991.768450629" Nov 01 14:50:53.493819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.493786 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:53.524790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.524765 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:50:53.565465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565440 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565507 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565530 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565665 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565585 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565673 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565717 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565768 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.565894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565826 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:53.565894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565853 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.566018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565917 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket\") pod \"2a714983-4771-43a1-b823-129a908d4254\" (UID: \"2a714983-4771-43a1-b823-129a908d4254\") " Nov 01 14:50:53.566018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.565933 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:53.566018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566000 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:50:53.566295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566269 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:53.566411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566389 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:53.566530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566508 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.566530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566531 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.566717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566547 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.566717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566562 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a714983-4771-43a1-b823-129a908d4254-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.566717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.566577 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.567793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.567771 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "kube-api-access-lwhv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:50:53.568011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.567978 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:50:53.568209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.568192 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:50:53.568296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.568191 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy" pod "2a714983-4771-43a1-b823-129a908d4254" (UID: "2a714983-4771-43a1-b823-129a908d4254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:50:53.601470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.601445 5473 generic.go:356] "Generic (PLEG): container finished" podID="2a714983-4771-43a1-b823-129a908d4254" containerID="69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc" exitCode=0 Nov 01 14:50:53.601570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.601490 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" Nov 01 14:50:53.601570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.601528 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" event={"ID":"2a714983-4771-43a1-b823-129a908d4254","Type":"ContainerDied","Data":"69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc"} Nov 01 14:50:53.601661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.601586 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z" event={"ID":"2a714983-4771-43a1-b823-129a908d4254","Type":"ContainerDied","Data":"a6c132777700c20a3ee486705fdf02cadc2f13e64c219b21ba4d9dd2006638a2"} Nov 01 14:50:53.601661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.601613 5473 scope.go:122] "RemoveContainer" containerID="69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc" Nov 01 14:50:53.611469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.611450 5473 scope.go:122] "RemoveContainer" containerID="9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133" Nov 01 14:50:53.620174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.620157 5473 scope.go:122] "RemoveContainer" containerID="69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc" Nov 01 14:50:53.620452 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:50:53.620432 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc\": container with ID starting with 69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc not found: ID does not exist" containerID="69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc" Nov 01 14:50:53.620536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.620458 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc"} err="failed to get container status \"69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc\": rpc error: code = NotFound desc = could not find container \"69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc\": container with ID starting with 69c68b8c9b6c5235d2b9d67034987c11fb497a785feb989e89e2be8efb080dfc not found: ID does not exist" Nov 01 14:50:53.620536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.620477 5473 scope.go:122] "RemoveContainer" containerID="9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133" Nov 01 14:50:53.620731 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:50:53.620714 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133\": container with ID starting with 9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133 not found: ID does not exist" containerID="9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133" Nov 01 14:50:53.620782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.620737 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133"} err="failed to get container status \"9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133\": rpc error: code = NotFound desc = could not find container \"9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133\": container with ID starting with 9938c11c65f8f748d49dac51a7aa71832f5a64cb689bb0f58a447418e778f133 not found: ID does not exist" Nov 01 14:50:53.623901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.623884 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:53.667823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.667801 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwhv4\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-kube-api-access-lwhv4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.667823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.667821 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a714983-4771-43a1-b823-129a908d4254-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.667955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.667833 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a714983-4771-43a1-b823-129a908d4254-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.667955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.667841 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a714983-4771-43a1-b823-129a908d4254-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:50:53.722583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.722549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:53.921029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.920997 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z"] Nov 01 14:50:53.930152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:53.928908 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c335a-openshift-default-5dd6bd6fcc-wnx2z"] Nov 01 14:50:54.610365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:54.610335 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:54.611127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:54.611107 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:50:55.081862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:55.081832 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a714983-4771-43a1-b823-129a908d4254" path="/var/lib/kubelet/pods/2a714983-4771-43a1-b823-129a908d4254/volumes" Nov 01 14:50:55.445277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:55.445183 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:55.689373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:55.689336 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:55.866406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:55.866374 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:57.784859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:57.784830 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:57.957152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:57.957048 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:58.584646 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:58.584605 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:58.746318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:58.746283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:50:59.874150 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:50:59.874121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:00.450916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:00.450885 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:00.930504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:00.930473 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:01.100127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:01.100081 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:01.109738 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:51:01.109709 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:51:01.497986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:01.497954 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:01.735908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:01.735866 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:02.348857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.348823 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:02.777241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.777196 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b"] Nov 01 14:51:02.777753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.777739 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" Nov 01 14:51:02.777797 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.777755 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" Nov 01 14:51:02.777797 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.777773 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" Nov 01 14:51:02.777797 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.777778 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" Nov 01 14:51:02.777910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.777858 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" Nov 01 14:51:02.781152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.781134 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.833238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.833188 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f764c7-openshift-default-dockercfg-rhh9m\"" Nov 01 14:51:02.886582 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.886550 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b"] Nov 01 14:51:02.893467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.893438 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:02.903934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.903903 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.903948 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904042 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njv44\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904081 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904165 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904257 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904335 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904488 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:02.904552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:02.904527 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005622 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005676 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005726 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njv44\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005766 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005805 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005859 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.005941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.005923 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.006205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.006079 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.006205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.006101 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.006205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.006101 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.006205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.006129 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.006522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.006500 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.006693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.006668 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.007051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.007026 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.008435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.008409 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.008812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.008792 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.062478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.062406 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.062610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.062567 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njv44\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44\") pod \"e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.094816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.094785 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:03.327093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.327007 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:03.521535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.520627 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b"] Nov 01 14:51:03.521535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.520943 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:03.521535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.521075 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:03.521535 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.521162 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:03.575681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.575655 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:03.672921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.672879 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" event={"ID":"300f8c59-371c-4990-8d63-48069e9f87fb","Type":"ContainerStarted","Data":"bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864"} Nov 01 14:51:03.673104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.672941 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" event={"ID":"300f8c59-371c-4990-8d63-48069e9f87fb","Type":"ContainerStarted","Data":"c23fc4f64a2f5c09e9374387549f3bb93183277238f19fc207754f901b99da90"} Nov 01 14:51:03.789081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:03.789028 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podStartSLOduration=1.789012877 podStartE2EDuration="1.789012877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:03.788124376 +0000 UTC m=+2002.846276238" watchObservedRunningTime="2026-11-01 14:51:03.789012877 +0000 UTC m=+2002.847164728" Nov 01 14:51:04.095846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:04.095812 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:04.100966 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:04.100939 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:04.679834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:04.679800 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:04.680812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:04.680793 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:51:06.042961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:06.042927 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:06.218416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:06.218381 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:06.619130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:06.619098 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:06.795785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:06.795730 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:06.959669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:06.959573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:08.391924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:08.391886 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:09.101449 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:09.101415 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:09.274984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:09.274950 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:13.051546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:13.051504 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:13.210536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:13.210503 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:13.368287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:13.368187 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:13.654476 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:13.654368 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:13.807401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:13.807367 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:14.019161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.019122 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm"] Nov 01 14:51:14.019875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.019853 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a714983-4771-43a1-b823-129a908d4254" containerName="istio-proxy" Nov 01 14:51:14.023793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.023774 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.076438 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.076405 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm"] Nov 01 14:51:14.122721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.122689 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f764c7-dockercfg-swzkm\"" Nov 01 14:51:14.123743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.123719 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f764c7\"" Nov 01 14:51:14.196746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.193496 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.196746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.193554 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrq8\" (UniqueName: \"kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8\") pod \"payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.294533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.294450 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.294533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.294499 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrq8\" (UniqueName: \"kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8\") pod \"payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.295129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.295109 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.355802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.355762 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrq8\" (UniqueName: \"kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8\") pod \"payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.365725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.365675 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:14.750992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.750902 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm"] Nov 01 14:51:14.755275 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:51:14.755245 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41870ba8_a6fb_405e_b6eb_4a80222a785f.slice/crio-fce3e845f0e5e6e0e29045a9096ca304e6ed14d0daf77d4710875c254e5b4817 WatchSource:0}: Error finding container fce3e845f0e5e6e0e29045a9096ca304e6ed14d0daf77d4710875c254e5b4817: Status 404 returned error can't find the container with id fce3e845f0e5e6e0e29045a9096ca304e6ed14d0daf77d4710875c254e5b4817 Nov 01 14:51:14.808810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:14.808785 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" event={"ID":"41870ba8-a6fb-405e-b6eb-4a80222a785f","Type":"ContainerStarted","Data":"fce3e845f0e5e6e0e29045a9096ca304e6ed14d0daf77d4710875c254e5b4817"} Nov 01 14:51:15.820840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:15.820802 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" event={"ID":"41870ba8-a6fb-405e-b6eb-4a80222a785f","Type":"ContainerStarted","Data":"19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d"} Nov 01 14:51:15.821339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:15.820907 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:15.959499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:15.959438 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" podStartSLOduration=2.9594221210000002 podStartE2EDuration="2.959422121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:51:15.957802213 +0000 UTC m=+2015.015954087" watchObservedRunningTime="2026-11-01 14:51:15.959422121 +0000 UTC m=+2015.017573971" Nov 01 14:51:18.015862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:18.015822 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:22.894377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:22.894342 5473 generic.go:356] "Generic (PLEG): container finished" podID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerID="c88fea85342ef19a2f5070940b366bb3dbe708a17bf40aa775f757d2622bbda5" exitCode=0 Nov 01 14:51:22.894957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:22.894429 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" event={"ID":"c8cbaae7-fb6f-468c-85a3-5d377df8e971","Type":"ContainerDied","Data":"c88fea85342ef19a2f5070940b366bb3dbe708a17bf40aa775f757d2622bbda5"} Nov 01 14:51:22.894957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:22.894849 5473 scope.go:122] "RemoveContainer" containerID="c88fea85342ef19a2f5070940b366bb3dbe708a17bf40aa775f757d2622bbda5" Nov 01 14:51:22.896472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:22.896447 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:22.896549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:22.896512 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:22.896588 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:22.896550 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:23.903813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:23.903772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:23.904470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:23.904451 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" event={"ID":"c8cbaae7-fb6f-468c-85a3-5d377df8e971","Type":"ContainerStarted","Data":"93295ae629a2b12f936ae03731f1e634474371e30207a8cd68e9b04f6ae45d0e"} Nov 01 14:51:23.944544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:23.944508 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:51:23.954268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:23.954227 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 14:51:23.954550 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:23.954518 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 14:51:24.947250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:24.946863 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 14:51:24.947250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:24.946922 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 14:51:25.115484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:25.115456 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:51:25.946241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:25.944599 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 14:51:25.946241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:25.944675 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 14:51:26.853889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:26.853857 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:51:26.947238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:26.946571 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 14:51:26.947238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:26.946642 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 14:51:27.949586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:27.949545 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 14:51:27.950052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:27.949603 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 14:51:28.250855 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:28.250766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:28.943176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:28.943134 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 01 14:51:28.943376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:28.943206 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 01 14:51:29.947756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:29.947725 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:51:29.948791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:29.948771 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:51:31.107776 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:51:31.107740 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:51:32.615082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:32.615045 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:37.060305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:37.060272 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:41.349225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:41.349179 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:41.521838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:41.521793 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:41.691279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:41.691190 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:42.629864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:42.629828 5473 scope.go:122] "RemoveContainer" containerID="f7a440f0933c560e31ecfc9955c16e649c0566024c24323635195209a664eda2" Nov 01 14:51:44.307062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:44.307024 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:45.106825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.106788 5473 generic.go:356] "Generic (PLEG): container finished" podID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerID="8c33ec8c4a63ed0bd11389d709d02dab94294cfd761c14e395dc8a58a504e036" exitCode=0 Nov 01 14:51:45.107064 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.106860 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" event={"ID":"f19b6ee9-20b0-4318-92ab-823cd52cb7dc","Type":"ContainerDied","Data":"8c33ec8c4a63ed0bd11389d709d02dab94294cfd761c14e395dc8a58a504e036"} Nov 01 14:51:45.107320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.107303 5473 scope.go:122] "RemoveContainer" containerID="8c33ec8c4a63ed0bd11389d709d02dab94294cfd761c14e395dc8a58a504e036" Nov 01 14:51:45.108851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.108826 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:45.108928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.108892 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:45.108928 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.108923 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:51:45.229457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.229417 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg"] Nov 01 14:51:45.324828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.324786 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh"] Nov 01 14:51:45.687899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:45.687836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:46.115030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:46.114994 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" event={"ID":"f19b6ee9-20b0-4318-92ab-823cd52cb7dc","Type":"ContainerStarted","Data":"ada24c9ecf682ec2fb2e3909ef8560878706457bbc54c0e90f402abdf8e0a1eb"} Nov 01 14:51:46.207804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:46.207768 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:51:46.207991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:46.207949 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 14:51:46.208042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:46.208011 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 14:51:47.205381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:47.205340 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:47.207356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:47.207334 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 14:51:47.207431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:47.207380 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 14:51:48.003883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:48.003848 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:48.207656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:48.207622 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 14:51:48.208088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:48.207675 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 14:51:49.208137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:49.208101 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 14:51:49.208601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:49.208155 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 14:51:50.207911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.207879 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 14:51:50.208104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.207934 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 14:51:50.247240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.246298 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" podUID="0dd2564b-117c-44c8-aa28-152ec30ef7b5" containerName="payload-processing" containerID="cri-o://660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6" gracePeriod=25 Nov 01 14:51:50.309022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.308990 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:50.342511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.342306 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" podUID="243c1e36-16c7-428f-a8be-09be0e50eb60" containerName="payload-pre-processing" containerID="cri-o://098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c" gracePeriod=25 Nov 01 14:51:50.587265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.586755 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:51:50.663206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.663171 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh\" (UniqueName: \"kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh\") pod \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " Nov 01 14:51:50.663377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.663313 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume\") pod \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\" (UID: \"0dd2564b-117c-44c8-aa28-152ec30ef7b5\") " Nov 01 14:51:50.663713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.663684 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume" pod "0dd2564b-117c-44c8-aa28-152ec30ef7b5" (UID: "0dd2564b-117c-44c8-aa28-152ec30ef7b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:51:50.663876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.663853 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dd2564b-117c-44c8-aa28-152ec30ef7b5-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:51:50.665467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.665437 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh" pod "0dd2564b-117c-44c8-aa28-152ec30ef7b5" (UID: "0dd2564b-117c-44c8-aa28-152ec30ef7b5"). InnerVolumeSpecName "kube-api-access-b2bwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:51:50.685080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.685053 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:51:50.765280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.765239 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw\" (UniqueName: \"kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw\") pod \"243c1e36-16c7-428f-a8be-09be0e50eb60\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " Nov 01 14:51:50.765478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.765294 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume\") pod \"243c1e36-16c7-428f-a8be-09be0e50eb60\" (UID: \"243c1e36-16c7-428f-a8be-09be0e50eb60\") " Nov 01 14:51:50.765790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.765762 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume" pod "243c1e36-16c7-428f-a8be-09be0e50eb60" (UID: "243c1e36-16c7-428f-a8be-09be0e50eb60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:51:50.765868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.765842 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2bwh\" (UniqueName: \"kubernetes.io/projected/0dd2564b-117c-44c8-aa28-152ec30ef7b5-kube-api-access-b2bwh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:51:50.767647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.767618 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw" pod "243c1e36-16c7-428f-a8be-09be0e50eb60" (UID: "243c1e36-16c7-428f-a8be-09be0e50eb60"). InnerVolumeSpecName "kube-api-access-mqbvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:51:50.867260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.867225 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqbvw\" (UniqueName: \"kubernetes.io/projected/243c1e36-16c7-428f-a8be-09be0e50eb60-kube-api-access-mqbvw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:51:50.867260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.867255 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/243c1e36-16c7-428f-a8be-09be0e50eb60-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:51:50.897603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:50.897569 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:51.171074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.171036 5473 generic.go:356] "Generic (PLEG): container finished" podID="243c1e36-16c7-428f-a8be-09be0e50eb60" containerID="098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c" exitCode=0 Nov 01 14:51:51.171279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.171092 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" Nov 01 14:51:51.171279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.171125 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" event={"ID":"243c1e36-16c7-428f-a8be-09be0e50eb60","Type":"ContainerDied","Data":"098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c"} Nov 01 14:51:51.171279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.171188 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh" event={"ID":"243c1e36-16c7-428f-a8be-09be0e50eb60","Type":"ContainerDied","Data":"cc0f0f777b7d4ff78055c5a29a745be305c2f3410e7bfd5156fb341e8ba82e28"} Nov 01 14:51:51.171279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.171239 5473 scope.go:122] "RemoveContainer" containerID="098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c" Nov 01 14:51:51.172827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.172802 5473 generic.go:356] "Generic (PLEG): container finished" podID="0dd2564b-117c-44c8-aa28-152ec30ef7b5" containerID="660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6" exitCode=0 Nov 01 14:51:51.172939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.172862 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" Nov 01 14:51:51.172939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.172883 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" event={"ID":"0dd2564b-117c-44c8-aa28-152ec30ef7b5","Type":"ContainerDied","Data":"660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6"} Nov 01 14:51:51.172939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.172932 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg" event={"ID":"0dd2564b-117c-44c8-aa28-152ec30ef7b5","Type":"ContainerDied","Data":"b92688fcb3e9f6de66d691062bd250a42b387cadfbfcad06408da9801ca2dc98"} Nov 01 14:51:51.184614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.184588 5473 scope.go:122] "RemoveContainer" containerID="098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c" Nov 01 14:51:51.185120 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:51:51.185097 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c\": container with ID starting with 098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c not found: ID does not exist" containerID="098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c" Nov 01 14:51:51.185198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.185129 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c"} err="failed to get container status \"098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c\": rpc error: code = NotFound desc = could not find container \"098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c\": container with ID starting with 098cc87b42cf80c1e1662756d7acecb17e63360e18e88a5d9e48538507b8120c not found: ID does not exist" Nov 01 14:51:51.185198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.185148 5473 scope.go:122] "RemoveContainer" containerID="660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6" Nov 01 14:51:51.194477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.194449 5473 scope.go:122] "RemoveContainer" containerID="660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6" Nov 01 14:51:51.194803 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:51:51.194785 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6\": container with ID starting with 660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6 not found: ID does not exist" containerID="660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6" Nov 01 14:51:51.194869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.194811 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6"} err="failed to get container status \"660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6\": rpc error: code = NotFound desc = could not find container \"660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6\": container with ID starting with 660fa1e3827e4cdea9a3b34f5df1e56f68b4c02fa4670e5136773968b863a1d6 not found: ID does not exist" Nov 01 14:51:51.208353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.208329 5473 patch_prober.go:29] interesting pod/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Nov 01 14:51:51.208479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.208386 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Nov 01 14:51:51.336830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.336798 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg"] Nov 01 14:51:51.373914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.373837 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-60547a-694db998fc-w8gkg"] Nov 01 14:51:51.452532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.452502 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh"] Nov 01 14:51:51.455411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:51.455388 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-60547a-6bcc958447-blnnh"] Nov 01 14:51:52.212386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:52.212356 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:51:52.212796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:52.212777 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:51:52.213550 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:52.213533 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:51:53.082049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:53.082005 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd2564b-117c-44c8-aa28-152ec30ef7b5" path="/var/lib/kubelet/pods/0dd2564b-117c-44c8-aa28-152ec30ef7b5/volumes" Nov 01 14:51:53.082692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:53.082669 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243c1e36-16c7-428f-a8be-09be0e50eb60" path="/var/lib/kubelet/pods/243c1e36-16c7-428f-a8be-09be0e50eb60/volumes" Nov 01 14:51:53.810931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:53.810902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:55.351249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:55.348650 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:56.405769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:56.405739 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:51:59.951564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:51:59.951533 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:00.114464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:00.114430 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:00.260923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:00.260838 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:01.109322 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:52:01.109292 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:52:01.239480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:01.239454 5473 generic.go:356] "Generic (PLEG): container finished" podID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerID="a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2" exitCode=0 Nov 01 14:52:01.239628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:01.239527 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" event={"ID":"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce","Type":"ContainerDied","Data":"a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2"} Nov 01 14:52:01.239974 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:01.239958 5473 scope.go:122] "RemoveContainer" containerID="a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2" Nov 01 14:52:01.241490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:01.241465 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:01.241547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:01.241531 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:01.241584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:01.241564 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:02.053834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.053792 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79"] Nov 01 14:52:02.102673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.102633 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ace39b-p6b79"] Nov 01 14:52:02.113819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.113787 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:02.256156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.256115 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" event={"ID":"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce","Type":"ContainerStarted","Data":"140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593"} Nov 01 14:52:02.279766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.279735 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:02.618536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.618505 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:52:02.618728 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.618704 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 14:52:02.618781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:02.618759 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 14:52:03.082525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:03.082494 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6143d1-38fa-4f4e-b2b7-f996f2135ccb" path="/var/lib/kubelet/pods/eb6143d1-38fa-4f4e-b2b7-f996f2135ccb/volumes" Nov 01 14:52:03.618693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:03.618659 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 14:52:03.619123 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:03.618716 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 14:52:04.475562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:04.475531 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:04.618402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:04.618366 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 14:52:04.618569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:04.618419 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 14:52:04.639275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:04.639180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:04.799691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:04.799656 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:05.618862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:05.618824 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 14:52:05.619043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:05.618886 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 14:52:06.618578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:06.618544 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 01 14:52:06.619100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:06.618596 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 01 14:52:07.622900 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:07.622862 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:52:07.623370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:07.623313 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:52:07.623992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:07.623961 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:52:09.097704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:09.097669 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:13.606981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:13.606941 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:17.447513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:17.447479 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:17.935627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:17.935596 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:18.097470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:18.097436 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:19.009698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:19.009667 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:19.258061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:19.258023 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:19.425422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:19.425384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:19.616211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:19.616177 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7"] Nov 01 14:52:19.800077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:19.800044 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:19.986194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:19.986160 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:20.162726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.162642 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:20.381965 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.381890 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7"] Nov 01 14:52:20.382466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.382439 5473 cpu_manager.go:414] "containerMap: removing container" podUID="243c1e36-16c7-428f-a8be-09be0e50eb60" containerName="payload-pre-processing" Nov 01 14:52:20.382466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.382468 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243c1e36-16c7-428f-a8be-09be0e50eb60" containerName="payload-pre-processing" Nov 01 14:52:20.382566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.382484 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0dd2564b-117c-44c8-aa28-152ec30ef7b5" containerName="payload-processing" Nov 01 14:52:20.382566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.382489 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd2564b-117c-44c8-aa28-152ec30ef7b5" containerName="payload-processing" Nov 01 14:52:20.382631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.382615 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="243c1e36-16c7-428f-a8be-09be0e50eb60" containerName="payload-pre-processing" Nov 01 14:52:20.382631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.382626 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd2564b-117c-44c8-aa28-152ec30ef7b5" containerName="payload-processing" Nov 01 14:52:20.386047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.386029 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.449731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.449644 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-288f1e-openshift-default-dockercfg-lthh4\"" Nov 01 14:52:20.468145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.468109 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7"] Nov 01 14:52:20.500287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.499368 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507513 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507590 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507627 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507680 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnh2\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507730 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507833 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507858 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.508164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.507903 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.603023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.602990 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:20.609524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.609462 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.611053 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.610952 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.611252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.611041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.611436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.611418 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.611577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.611560 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.614825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.611422 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.615101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.615080 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnh2\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.615721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.615694 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.615826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.615774 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.615826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.615801 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.615940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.615845 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.616035 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.615426 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.616412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.616392 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.617077 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.617054 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.624257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.619748 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.624257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.622682 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.685240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.673890 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.689666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.685874 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnh2\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2\") pod \"e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.709282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.707565 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:20.822232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:20.822157 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:21.001633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.001559 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:21.210317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.210271 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7"] Nov 01 14:52:21.279372 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.279333 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:21.279562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.279406 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:21.279562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.279439 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:21.285282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.283424 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf"] Nov 01 14:52:21.285282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.283793 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" containerID="cri-o://92193297149cd4434e5ac669201f1b8abbdd6415c2cd182a9418fb39e4af5f79" gracePeriod=30 Nov 01 14:52:21.305398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.305367 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:21.486804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.486773 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" event={"ID":"61ed7157-72cf-49b8-b90c-a01470139213","Type":"ContainerStarted","Data":"e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08"} Nov 01 14:52:21.486941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.486821 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" event={"ID":"61ed7157-72cf-49b8-b90c-a01470139213","Type":"ContainerStarted","Data":"f0e80f4b4675db9805a9d4592b029a044cd882a164ab2a724b906baca8551cba"} Nov 01 14:52:21.619922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.619868 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podStartSLOduration=1.619855606 podStartE2EDuration="1.619855606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:21.617813671 +0000 UTC m=+2080.675965522" watchObservedRunningTime="2026-11-01 14:52:21.619855606 +0000 UTC m=+2080.678007519" Nov 01 14:52:21.708691 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:21.708657 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:22.055633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:22.055595 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:22.714199 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:22.714168 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:23.499242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:23.499180 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:23.500166 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:23.500139 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:52:24.320909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:24.320874 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:24.617407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:24.617324 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" podUID="61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" containerName="payload-processing" containerID="cri-o://b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230" gracePeriod=25 Nov 01 14:52:24.880697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:24.880111 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:24.915173 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:24.915152 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:52:25.017554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.017520 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume\") pod \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " Nov 01 14:52:25.017737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.017719 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h\" (UniqueName: \"kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h\") pod \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\" (UID: \"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1\") " Nov 01 14:52:25.018041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.018001 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume" pod "61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" (UID: "61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:52:25.018300 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.018282 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:25.021715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.021693 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h" pod "61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" (UID: "61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1"). InnerVolumeSpecName "kube-api-access-fkh7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:52:25.064909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.064866 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:25.119841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.119809 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkh7h\" (UniqueName: \"kubernetes.io/projected/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1-kube-api-access-fkh7h\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:25.250481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.250401 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:25.513409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.513367 5473 generic.go:356] "Generic (PLEG): container finished" podID="61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" containerID="b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230" exitCode=0 Nov 01 14:52:25.513825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.513414 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" Nov 01 14:52:25.513825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.513455 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" event={"ID":"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1","Type":"ContainerDied","Data":"b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230"} Nov 01 14:52:25.513825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.513600 5473 scope.go:122] "RemoveContainer" containerID="b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230" Nov 01 14:52:25.513825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.513697 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7" event={"ID":"61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1","Type":"ContainerDied","Data":"9a3f4a92cae97b52b930a1cc0fc2733ee9bcf11345e41ab48da59a2fb8654057"} Nov 01 14:52:25.554453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.545192 5473 scope.go:122] "RemoveContainer" containerID="b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230" Nov 01 14:52:25.554453 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:52:25.546651 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230\": container with ID starting with b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230 not found: ID does not exist" containerID="b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230" Nov 01 14:52:25.554453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.546688 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230"} err="failed to get container status \"b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230\": rpc error: code = NotFound desc = could not find container \"b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230\": container with ID starting with b71ad03119b465cff823e438d8d665b9fdc5e4bff60ced361fc2705de9020230 not found: ID does not exist" Nov 01 14:52:25.679139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.679107 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7"] Nov 01 14:52:25.750769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:25.750722 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2048c1-d98dd7965-ph4z7"] Nov 01 14:52:26.523349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.523306 5473 generic.go:356] "Generic (PLEG): container finished" podID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerID="92193297149cd4434e5ac669201f1b8abbdd6415c2cd182a9418fb39e4af5f79" exitCode=0 Nov 01 14:52:26.523761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.523372 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" event={"ID":"7e9d24b2-93c6-4e08-b2a0-f7841567a71a","Type":"ContainerDied","Data":"92193297149cd4434e5ac669201f1b8abbdd6415c2cd182a9418fb39e4af5f79"} Nov 01 14:52:26.610439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.610416 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:52:26.663445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663412 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.663698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663480 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.663698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663502 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.663698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663524 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.663698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663577 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.663698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663605 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.663698 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663701 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.664084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663758 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.664084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663808 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs\") pod \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\" (UID: \"7e9d24b2-93c6-4e08-b2a0-f7841567a71a\") " Nov 01 14:52:26.664084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.663897 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:26.664262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.664127 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:26.664396 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.664372 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:26.664769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.664612 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.664769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.664637 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.664769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.664655 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.664769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.664741 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:52:26.665135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.665117 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:26.665964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.665942 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:52:26.666774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.666757 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "kube-api-access-gwpwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:52:26.666913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.666889 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:26.666984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.666893 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo" pod "7e9d24b2-93c6-4e08-b2a0-f7841567a71a" (UID: "7e9d24b2-93c6-4e08-b2a0-f7841567a71a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:52:26.765687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.765643 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpwt\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-kube-api-access-gwpwt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.765687 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.765683 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.765937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.765700 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.765937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.765714 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.765937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.765729 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:26.765937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:26.765743 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e9d24b2-93c6-4e08-b2a0-f7841567a71a-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:27.082143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:27.082106 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" path="/var/lib/kubelet/pods/61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1/volumes" Nov 01 14:52:27.554580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:27.554538 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" event={"ID":"7e9d24b2-93c6-4e08-b2a0-f7841567a71a","Type":"ContainerDied","Data":"8dbea7f99d985f53b0ee9ecfe8ee8eadd4016e75a2345e9d0dcd6504e69f512d"} Nov 01 14:52:27.555040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:27.554594 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf" Nov 01 14:52:27.555040 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:27.554619 5473 scope.go:122] "RemoveContainer" containerID="92193297149cd4434e5ac669201f1b8abbdd6415c2cd182a9418fb39e4af5f79" Nov 01 14:52:27.780656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:27.780612 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf"] Nov 01 14:52:27.782412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:27.782387 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-34d384-openshift-default-5d9f84bc87-65ftf"] Nov 01 14:52:28.041893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.041860 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:28.214010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.213979 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:28.382394 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.382314 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:28.419135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419105 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb"] Nov 01 14:52:28.419670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419656 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" Nov 01 14:52:28.419744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419672 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" Nov 01 14:52:28.419744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419694 5473 cpu_manager.go:414] "containerMap: removing container" podUID="61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" containerName="payload-processing" Nov 01 14:52:28.419744 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419700 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" containerName="payload-processing" Nov 01 14:52:28.419843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419787 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" containerName="istio-proxy" Nov 01 14:52:28.419843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.419798 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cd6e5c-7a1d-4a8b-a89e-a24d44072ef1" containerName="payload-processing" Nov 01 14:52:28.424634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.424617 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.477185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.477157 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-288f1e\"" Nov 01 14:52:28.477487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.477455 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb"] Nov 01 14:52:28.477674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.477595 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-288f1e-dockercfg-rnfsm\"" Nov 01 14:52:28.518366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.518326 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9457\" (UniqueName: \"kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.523272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.518569 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.585613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.585579 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx"] Nov 01 14:52:28.590447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.590418 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.619378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.619339 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.619565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.619405 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9457\" (UniqueName: \"kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.620027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.620003 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.640225 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.640120 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx"] Nov 01 14:52:28.676730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.676697 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9457\" (UniqueName: \"kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457\") pod \"payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.720207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.720175 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.720410 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.720298 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d456v\" (UniqueName: \"kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v\") pod \"payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.739085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.739045 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:28.821669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.821633 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.821854 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.821719 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d456v\" (UniqueName: \"kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v\") pod \"payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.822681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.822638 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.878875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.878846 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d456v\" (UniqueName: \"kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v\") pod \"payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:28.904983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:28.904902 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:29.083354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.083315 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9d24b2-93c6-4e08-b2a0-f7841567a71a" path="/var/lib/kubelet/pods/7e9d24b2-93c6-4e08-b2a0-f7841567a71a/volumes" Nov 01 14:52:29.111863 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:52:29.111822 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c193a1b_61c1_45dd_8135_f7c3f13640a9.slice/crio-b858cac8b9c5a580f8de606d63b33b0824805fcd414a3b7f4c1c95ef01276123 WatchSource:0}: Error finding container b858cac8b9c5a580f8de606d63b33b0824805fcd414a3b7f4c1c95ef01276123: Status 404 returned error can't find the container with id b858cac8b9c5a580f8de606d63b33b0824805fcd414a3b7f4c1c95ef01276123 Nov 01 14:52:29.114423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.114393 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb"] Nov 01 14:52:29.216683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.216651 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:29.278134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.278102 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx"] Nov 01 14:52:29.278288 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:52:29.278181 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce38177d_574e_4ad8_b159_0c0df99fd860.slice/crio-260d457b0a4e3dbc4c9176543f331e68aaa233c6bc5867d4e4b3739c11772b18 WatchSource:0}: Error finding container 260d457b0a4e3dbc4c9176543f331e68aaa233c6bc5867d4e4b3739c11772b18: Status 404 returned error can't find the container with id 260d457b0a4e3dbc4c9176543f331e68aaa233c6bc5867d4e4b3739c11772b18 Nov 01 14:52:29.593743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.593704 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" event={"ID":"ce38177d-574e-4ad8-b159-0c0df99fd860","Type":"ContainerStarted","Data":"611d353d39c4660b9a5b6f32bea48ae2847020b4779796c7031d5010be33ce63"} Nov 01 14:52:29.594358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.594313 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" event={"ID":"ce38177d-574e-4ad8-b159-0c0df99fd860","Type":"ContainerStarted","Data":"260d457b0a4e3dbc4c9176543f331e68aaa233c6bc5867d4e4b3739c11772b18"} Nov 01 14:52:29.594487 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.594367 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:29.595630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.595606 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" event={"ID":"4c193a1b-61c1-45dd-8135-f7c3f13640a9","Type":"ContainerStarted","Data":"60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792"} Nov 01 14:52:29.595779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.595687 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:29.595779 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.595706 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" event={"ID":"4c193a1b-61c1-45dd-8135-f7c3f13640a9","Type":"ContainerStarted","Data":"b858cac8b9c5a580f8de606d63b33b0824805fcd414a3b7f4c1c95ef01276123"} Nov 01 14:52:29.709263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.709178 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" podStartSLOduration=1.7091634679999999 podStartE2EDuration="1.709163468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:29.706889676 +0000 UTC m=+2088.765041526" watchObservedRunningTime="2026-11-01 14:52:29.709163468 +0000 UTC m=+2088.767315317" Nov 01 14:52:29.835238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:29.832512 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" podStartSLOduration=1.8324941510000001 podStartE2EDuration="1.832494151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:29.821809965 +0000 UTC m=+2088.879961834" watchObservedRunningTime="2026-11-01 14:52:29.832494151 +0000 UTC m=+2088.890646001" Nov 01 14:52:30.603474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:30.603441 5473 generic.go:356] "Generic (PLEG): container finished" podID="300f8c59-371c-4990-8d63-48069e9f87fb" containerID="bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864" exitCode=0 Nov 01 14:52:30.603904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:30.603499 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" event={"ID":"300f8c59-371c-4990-8d63-48069e9f87fb","Type":"ContainerDied","Data":"bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864"} Nov 01 14:52:30.604123 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:30.604105 5473 scope.go:122] "RemoveContainer" containerID="bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864" Nov 01 14:52:30.605508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:30.605483 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:30.605598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:30.605545 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:30.605598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:30.605574 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:31.123672 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:52:31.123625 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:52:31.613427 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:31.613394 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" event={"ID":"300f8c59-371c-4990-8d63-48069e9f87fb","Type":"ContainerStarted","Data":"a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4"} Nov 01 14:52:32.095879 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:32.095844 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:52:32.096817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:32.096787 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 14:52:32.096954 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:32.096850 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 14:52:33.095507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:33.095473 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 14:52:33.095909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:33.095536 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 14:52:33.462557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:33.462468 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:34.095152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:34.095120 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 14:52:34.095368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:34.095187 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 14:52:34.680495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:34.680457 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:52:35.095185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:35.095145 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 14:52:35.095422 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:35.095202 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 14:52:35.375761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:35.375663 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:36.095205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:36.095160 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 01 14:52:36.095645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:36.095240 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 01 14:52:37.099972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:37.099942 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:52:37.100930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:37.100910 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:52:40.105990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:40.105953 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:40.607298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:40.607269 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:52:41.617081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:41.617052 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:52:42.415282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:42.415244 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:42.740151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:42.740114 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:42.907951 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:42.907896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:43.546376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:43.546337 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:43.774113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:43.774085 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:52:43.777546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:43.777521 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:52:44.123036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.123000 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:44.287383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.287342 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:44.479248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.479158 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:44.622803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.622766 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm"] Nov 01 14:52:44.622999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.622977 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" containerID="cri-o://93295ae629a2b12f936ae03731f1e634474371e30207a8cd68e9b04f6ae45d0e" gracePeriod=30 Nov 01 14:52:44.645710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.645676 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:44.950842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.950806 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:52:44.951270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:44.950857 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:52:45.088170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:45.088137 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:46.085895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:46.085859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:46.972205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:46.972173 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw"] Nov 01 14:52:46.977506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:46.977491 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.030519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.030484 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a90080-openshift-default-dockercfg-87tlq\"" Nov 01 14:52:47.081313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.081271 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:47.086182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.086134 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw"] Nov 01 14:52:47.104942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.104914 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.104988 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105133 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhn5\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105178 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105313 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105294 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105340 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105501 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105550 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.105642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.105622 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.206653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206620 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.206832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206677 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.206832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206716 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.206832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206746 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhn5\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.206832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206765 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206841 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.206896 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207036 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207091 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207146 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207289 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207403 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207502 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.207755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.207731 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.209185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.209164 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.209331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.209312 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.263741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.263710 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.263858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.263834 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhn5\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5\") pod \"e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.291174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.291153 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:47.458808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.458779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:47.711746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.711667 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw"] Nov 01 14:52:47.742928 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:52:47.742899 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d536e10_e370_47e1_a3ed_122c1b806f5b.slice/crio-5a6a8cd838d1f9571a6a38d8ee80f0aa3fccbd5ecde229359bdb34b0d0c21849 WatchSource:0}: Error finding container 5a6a8cd838d1f9571a6a38d8ee80f0aa3fccbd5ecde229359bdb34b0d0c21849: Status 404 returned error can't find the container with id 5a6a8cd838d1f9571a6a38d8ee80f0aa3fccbd5ecde229359bdb34b0d0c21849 Nov 01 14:52:47.745416 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.745382 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:47.745519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.745446 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:47.745519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.745474 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:52:47.767444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.767422 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" event={"ID":"7d536e10-e370-47e1-a3ed-122c1b806f5b","Type":"ContainerStarted","Data":"5a6a8cd838d1f9571a6a38d8ee80f0aa3fccbd5ecde229359bdb34b0d0c21849"} Nov 01 14:52:47.928523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:47.928480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:48.106594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:48.106565 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:48.776049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:48.776007 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" event={"ID":"7d536e10-e370-47e1-a3ed-122c1b806f5b","Type":"ContainerStarted","Data":"25d032ee285c78272d152a9c19fa08bb38e1f5c3ce1ace6eeacc5e57330da4e3"} Nov 01 14:52:48.894880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:48.894822 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podStartSLOduration=2.894803118 podStartE2EDuration="2.894803118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:48.892610695 +0000 UTC m=+2107.950762545" watchObservedRunningTime="2026-11-01 14:52:48.894803118 +0000 UTC m=+2107.952955004" Nov 01 14:52:49.292014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.291986 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:49.297730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.297706 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:49.785352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.785322 5473 generic.go:356] "Generic (PLEG): container finished" podID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerID="93295ae629a2b12f936ae03731f1e634474371e30207a8cd68e9b04f6ae45d0e" exitCode=0 Nov 01 14:52:49.785513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.785412 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" event={"ID":"c8cbaae7-fb6f-468c-85a3-5d377df8e971","Type":"ContainerDied","Data":"93295ae629a2b12f936ae03731f1e634474371e30207a8cd68e9b04f6ae45d0e"} Nov 01 14:52:49.785513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.785486 5473 scope.go:122] "RemoveContainer" containerID="c88fea85342ef19a2f5070940b366bb3dbe708a17bf40aa775f757d2622bbda5" Nov 01 14:52:49.785801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.785704 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:49.786800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.786782 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:52:49.853865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.853796 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:49.895036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:49.894956 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:52:50.042159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042124 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042262 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042285 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042317 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042385 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042417 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:50.042564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042431 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042488 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042561 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.042842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.042635 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert\") pod \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\" (UID: \"c8cbaae7-fb6f-468c-85a3-5d377df8e971\") " Nov 01 14:52:50.043163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.043137 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.043562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.043327 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:50.043562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.043423 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:50.043761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.043689 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:52:50.043910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.043877 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:50.044754 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.044732 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "kube-api-access-d2b2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:52:50.044973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.044957 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:52:50.045144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.045126 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:52:50.045451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.045425 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token" pod "c8cbaae7-fb6f-468c-85a3-5d377df8e971" (UID: "c8cbaae7-fb6f-468c-85a3-5d377df8e971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:52:50.144944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144916 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.144944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144943 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.145157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144953 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.145157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144963 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.145157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144972 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.145157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144980 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.145157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144989 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cbaae7-fb6f-468c-85a3-5d377df8e971-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.145157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.144999 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2b2q\" (UniqueName: \"kubernetes.io/projected/c8cbaae7-fb6f-468c-85a3-5d377df8e971-kube-api-access-d2b2q\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:52:50.793268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.793240 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" Nov 01 14:52:50.793688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.793266 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm" event={"ID":"c8cbaae7-fb6f-468c-85a3-5d377df8e971","Type":"ContainerDied","Data":"79974231676a3cde85fa874ff0472f0e1127a6e1f479881b32e050827db0c007"} Nov 01 14:52:50.793688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.793329 5473 scope.go:122] "RemoveContainer" containerID="93295ae629a2b12f936ae03731f1e634474371e30207a8cd68e9b04f6ae45d0e" Nov 01 14:52:50.958632 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.958594 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm"] Nov 01 14:52:50.962056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:50.962032 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-60547a-openshift-default-7d944df55f-nx8sm"] Nov 01 14:52:51.001395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.001366 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:51.084675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.084595 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" path="/var/lib/kubelet/pods/c8cbaae7-fb6f-468c-85a3-5d377df8e971/volumes" Nov 01 14:52:51.171234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.171153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:51.362470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.362394 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7"] Nov 01 14:52:51.362929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.362916 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" Nov 01 14:52:51.362970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.362930 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" Nov 01 14:52:51.362970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.362960 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" Nov 01 14:52:51.362970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.362966 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" Nov 01 14:52:51.363070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.363040 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" Nov 01 14:52:51.363070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.363052 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cbaae7-fb6f-468c-85a3-5d377df8e971" containerName="istio-proxy" Nov 01 14:52:51.367867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.367848 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.419249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.419187 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7"] Nov 01 14:52:51.420517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.420492 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-305290\"" Nov 01 14:52:51.420664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.420641 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-305290-dockercfg-2kbfk\"" Nov 01 14:52:51.462270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.462194 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbkr\" (UniqueName: \"kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr\") pod \"payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.462749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.462725 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.533501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.533455 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm"] Nov 01 14:52:51.540515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.540490 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.563768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.563742 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw96t\" (UniqueName: \"kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t\") pod \"payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.563902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.563799 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.563955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.563917 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbkr\" (UniqueName: \"kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr\") pod \"payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.564084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.564067 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.564553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.564534 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.597890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.597863 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm"] Nov 01 14:52:51.621271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.621193 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbkr\" (UniqueName: \"kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr\") pod \"payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.664940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.664910 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.665093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.664969 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw96t\" (UniqueName: \"kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t\") pod \"payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.665621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.665599 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.681641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.681618 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:51.721904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.721872 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw96t\" (UniqueName: \"kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t\") pod \"payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:51.856656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:51.856624 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:52.053568 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:52:52.053521 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f51451_c201_4896_90ef_5e34d74e59ae.slice/crio-48013fb83f8ec91d303ad76c00d1fc9daab8d54c281740741fe89535cd9463b9 WatchSource:0}: Error finding container 48013fb83f8ec91d303ad76c00d1fc9daab8d54c281740741fe89535cd9463b9: Status 404 returned error can't find the container with id 48013fb83f8ec91d303ad76c00d1fc9daab8d54c281740741fe89535cd9463b9 Nov 01 14:52:52.057680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.057656 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7"] Nov 01 14:52:52.224319 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:52:52.224290 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f54644_cc0e_40fc_98ca_4a122472ed7f.slice/crio-c9d4f77d1586e687443ffe8e43791f4c27f9a1e5d86e675a250c90776065b985 WatchSource:0}: Error finding container c9d4f77d1586e687443ffe8e43791f4c27f9a1e5d86e675a250c90776065b985: Status 404 returned error can't find the container with id c9d4f77d1586e687443ffe8e43791f4c27f9a1e5d86e675a250c90776065b985 Nov 01 14:52:52.228032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.228009 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm"] Nov 01 14:52:52.479108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.479026 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:52.809066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.809030 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" event={"ID":"65f54644-cc0e-40fc-98ca-4a122472ed7f","Type":"ContainerStarted","Data":"95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b"} Nov 01 14:52:52.809339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.809098 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:52:52.809339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.809126 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" event={"ID":"65f54644-cc0e-40fc-98ca-4a122472ed7f","Type":"ContainerStarted","Data":"c9d4f77d1586e687443ffe8e43791f4c27f9a1e5d86e675a250c90776065b985"} Nov 01 14:52:52.810901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.810879 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" event={"ID":"80f51451-c201-4896-90ef-5e34d74e59ae","Type":"ContainerStarted","Data":"7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c"} Nov 01 14:52:52.811015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.810917 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" event={"ID":"80f51451-c201-4896-90ef-5e34d74e59ae","Type":"ContainerStarted","Data":"48013fb83f8ec91d303ad76c00d1fc9daab8d54c281740741fe89535cd9463b9"} Nov 01 14:52:52.811015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.810970 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:52:52.839963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.839940 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:52.933186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:52.933133 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" podStartSLOduration=1.933119704 podStartE2EDuration="1.933119704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:52.930202742 +0000 UTC m=+2111.988354591" watchObservedRunningTime="2026-11-01 14:52:52.933119704 +0000 UTC m=+2111.991271553" Nov 01 14:52:53.065543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:53.065347 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" podStartSLOduration=2.065327962 podStartE2EDuration="2.065327962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:52:53.063356375 +0000 UTC m=+2112.121508227" watchObservedRunningTime="2026-11-01 14:52:53.065327962 +0000 UTC m=+2112.123479813" Nov 01 14:52:54.136926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:54.136893 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:54.822716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:54.822686 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:55.627356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:55.627326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:55.783816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:55.783783 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:52:59.407537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:52:59.407507 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:01.107477 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:53:01.107445 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:53:03.820306 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:03.820278 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:53:03.820747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:03.820343 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:53:04.192675 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:04.192595 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:04.358359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:04.358326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:07.718153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:07.718116 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:08.886763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:08.886729 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:09.300822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:09.300792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:10.025957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.025924 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:10.470390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.470312 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w"] Nov 01 14:53:10.474479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.474458 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.527189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.527158 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5223a8-openshift-default-dockercfg-5cqqc\"" Nov 01 14:53:10.580339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.580308 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w"] Nov 01 14:53:10.585653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.585609 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.585891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.585870 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.586030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.586014 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.586149 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.586134 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrjhc\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.586364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.586346 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.589769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.586823 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.589769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.587068 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.589769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.587148 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.589769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.587194 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.599168 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.596566 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:10.688539 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688502 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688548 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688574 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688594 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrjhc\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688633 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688684 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688758 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688790 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.688893 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688815 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.689085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688923 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.689085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.688949 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.689085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.689041 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.689197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.689173 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.689669 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.689652 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.691003 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.690982 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.691132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.691115 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.782858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.782818 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrjhc\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.783897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.783867 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token\") pod \"e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:10.788175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.788123 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:10.789157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:10.789113 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:11.210339 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:53:11.210309 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0c815e_4363_4ce1_ad46_bccbe276941b.slice/crio-13068b746bdb5fe3502c6cd83f0c449863531addb4fb7485442b427297f949dd WatchSource:0}: Error finding container 13068b746bdb5fe3502c6cd83f0c449863531addb4fb7485442b427297f949dd: Status 404 returned error can't find the container with id 13068b746bdb5fe3502c6cd83f0c449863531addb4fb7485442b427297f949dd Nov 01 14:53:11.213404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:11.213364 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:53:11.213513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:11.213448 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:53:11.213513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:11.213489 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:53:11.213904 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:11.213880 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w"] Nov 01 14:53:11.942876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:11.942838 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" event={"ID":"ff0c815e-4363-4ce1-ad46-bccbe276941b","Type":"ContainerStarted","Data":"4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1"} Nov 01 14:53:11.943106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:11.942905 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" event={"ID":"ff0c815e-4363-4ce1-ad46-bccbe276941b","Type":"ContainerStarted","Data":"13068b746bdb5fe3502c6cd83f0c449863531addb4fb7485442b427297f949dd"} Nov 01 14:53:12.061663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:12.061598 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" podStartSLOduration=2.061583689 podStartE2EDuration="2.061583689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:12.059249904 +0000 UTC m=+2131.117401756" watchObservedRunningTime="2026-11-01 14:53:12.061583689 +0000 UTC m=+2131.119735542" Nov 01 14:53:12.790038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:12.790007 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:12.796447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:12.796419 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:12.887045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:12.887003 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:12.974089 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:12.974055 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:12.975630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:12.975608 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:53:13.370995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:13.370964 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:13.541237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:13.537720 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:13.726294 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:13.725627 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:14.155193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:14.155160 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:15.009367 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:15.009332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:15.179298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:15.179269 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:15.354198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:15.354167 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:15.998810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:15.998781 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:18.397033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.396995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:18.596001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.595966 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:18.773228 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.773189 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t"] Nov 01 14:53:18.777516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.777499 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.822799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.822775 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.823069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.823046 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9b9m\" (UniqueName: \"kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m\") pod \"payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.829153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.829125 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t"] Nov 01 14:53:18.880243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.876498 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5223a8\"" Nov 01 14:53:18.895347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.880424 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5223a8-dockercfg-6ch95\"" Nov 01 14:53:18.925195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.925160 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9b9m\" (UniqueName: \"kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m\") pod \"payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.925874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.925850 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.936463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.936432 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.940817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.940789 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj"] Nov 01 14:53:18.945324 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.945302 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:18.982493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.982470 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9b9m\" (UniqueName: \"kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m\") pod \"payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:18.992005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:18.991981 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj"] Nov 01 14:53:19.031315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.031204 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.031474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.031458 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wms4\" (UniqueName: \"kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4\") pod \"payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.091945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.091918 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:19.132846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.132593 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wms4\" (UniqueName: \"kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4\") pod \"payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.132846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.132689 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.133832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.133806 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.190856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.190812 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wms4\" (UniqueName: \"kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4\") pod \"payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.259642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.259610 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:19.462208 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:53:19.462175 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9c832a_f5ed_4fc1_9e17_f41f9331d982.slice/crio-5e72de5c4e03567eed0dd6a2447509fdb16466d327ee2c6929f09acecdf96e62 WatchSource:0}: Error finding container 5e72de5c4e03567eed0dd6a2447509fdb16466d327ee2c6929f09acecdf96e62: Status 404 returned error can't find the container with id 5e72de5c4e03567eed0dd6a2447509fdb16466d327ee2c6929f09acecdf96e62 Nov 01 14:53:19.465711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.465685 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t"] Nov 01 14:53:19.630840 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:53:19.630812 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe70667d_623d_4e12_b808_bb1ab138dab8.slice/crio-9e8a2901104404fb6bb41c2cfb7990f7c15cef54d02bbb799b95542878ba1c35 WatchSource:0}: Error finding container 9e8a2901104404fb6bb41c2cfb7990f7c15cef54d02bbb799b95542878ba1c35: Status 404 returned error can't find the container with id 9e8a2901104404fb6bb41c2cfb7990f7c15cef54d02bbb799b95542878ba1c35 Nov 01 14:53:19.634070 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.634048 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj"] Nov 01 14:53:19.662236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:19.662198 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:20.039043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.039009 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" event={"ID":"ba9c832a-f5ed-4fc1-9e17-f41f9331d982","Type":"ContainerStarted","Data":"fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375"} Nov 01 14:53:20.039257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.039069 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" event={"ID":"ba9c832a-f5ed-4fc1-9e17-f41f9331d982","Type":"ContainerStarted","Data":"5e72de5c4e03567eed0dd6a2447509fdb16466d327ee2c6929f09acecdf96e62"} Nov 01 14:53:20.039257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.039191 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:20.040791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.040768 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" event={"ID":"fe70667d-623d-4e12-b808-bb1ab138dab8","Type":"ContainerStarted","Data":"56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d"} Nov 01 14:53:20.040898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.040807 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" event={"ID":"fe70667d-623d-4e12-b808-bb1ab138dab8","Type":"ContainerStarted","Data":"9e8a2901104404fb6bb41c2cfb7990f7c15cef54d02bbb799b95542878ba1c35"} Nov 01 14:53:20.040898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.040860 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:20.154125 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.154067 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" podStartSLOduration=2.154054971 podStartE2EDuration="2.154054971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:20.152444998 +0000 UTC m=+2139.210596847" watchObservedRunningTime="2026-11-01 14:53:20.154054971 +0000 UTC m=+2139.212206821" Nov 01 14:53:20.273793 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:20.272035 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" podStartSLOduration=2.272016497 podStartE2EDuration="2.272016497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:53:20.267045658 +0000 UTC m=+2139.325197521" watchObservedRunningTime="2026-11-01 14:53:20.272016497 +0000 UTC m=+2139.330168348" Nov 01 14:53:21.057341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:21.057307 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:21.266044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:21.266010 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:21.417891 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:21.417813 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:25.792677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:25.792642 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:30.243598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:30.243534 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:31.051589 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:31.051561 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:53:31.051768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:31.051625 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:53:31.108194 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:53:31.108161 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:53:34.710662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:34.710628 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:38.029923 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:38.029883 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:38.554263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:38.554236 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:40.247364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:40.247327 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:42.696553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:42.696521 5473 scope.go:122] "RemoveContainer" containerID="52fb0ef4ada4c3c432b71487bf6cd26d73ab0263348dbdcb0421de13b13c3a98" Nov 01 14:53:42.782333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:42.782301 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:42.943352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:42.943319 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:43.048488 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:43.048450 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk"] Nov 01 14:53:43.100737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:43.100702 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db15f32-wdsxk"] Nov 01 14:53:44.278872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:44.278841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:44.862192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:44.862159 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:45.037616 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:45.037581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:45.082658 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:45.082625 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f834c040-a487-468d-9806-e517879c1675" path="/var/lib/kubelet/pods/f834c040-a487-468d-9806-e517879c1675/volumes" Nov 01 14:53:45.491275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:45.491243 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:49.705697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:49.705659 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:49.879831 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:49.879792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:50.050061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:50.050017 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:51.654849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:51.654814 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:53.810877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:53.809464 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": read tcp 10.134.0.2:41052->10.134.0.170:15021: read: connection reset by peer" start-of-body= Nov 01 14:53:53.810877 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:53.809526 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": read tcp 10.134.0.2:41052->10.134.0.170:15021: read: connection reset by peer" Nov 01 14:53:54.031282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.031251 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:54.193447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.193365 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:54.314399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.314367 5473 generic.go:356] "Generic (PLEG): container finished" podID="61ed7157-72cf-49b8-b90c-a01470139213" containerID="e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08" exitCode=0 Nov 01 14:53:54.314559 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.314442 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" event={"ID":"61ed7157-72cf-49b8-b90c-a01470139213","Type":"ContainerDied","Data":"e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08"} Nov 01 14:53:54.314853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.314833 5473 scope.go:122] "RemoveContainer" containerID="e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08" Nov 01 14:53:54.316327 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.316295 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:53:54.316451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.316384 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:53:54.316451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.316434 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:53:54.470716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.470614 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:54.917786 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:54.917753 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:55.322697 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:55.322662 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" event={"ID":"61ed7157-72cf-49b8-b90c-a01470139213","Type":"ContainerStarted","Data":"3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0"} Nov 01 14:53:55.709285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:55.709184 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:53:55.709439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:55.709394 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 14:53:55.709489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:55.709443 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 14:53:56.709113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:56.709078 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 14:53:56.709568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:56.709133 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 14:53:57.350615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:57.350581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:57.708732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:57.708648 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 14:53:57.708732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:57.708703 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 14:53:58.708988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:58.708950 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 14:53:58.709414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:58.709012 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 14:53:59.150627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:59.150594 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:53:59.708398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:59.708364 5473 patch_prober.go:29] interesting pod/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Nov 01 14:53:59.708576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:53:59.708419 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Nov 01 14:54:00.712092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:00.712064 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:54:00.712624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:00.712513 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:54:00.713207 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:00.713190 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:54:00.955360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:00.955326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:01.108245 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:54:01.108188 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:54:03.043886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:03.043850 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:03.211064 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:03.211031 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:04.467456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:04.467426 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:04.962055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:04.962024 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:05.564864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:05.564834 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:05.965374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:05.965292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:10.971369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:10.971314 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:13.494947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:13.494916 5473 generic.go:356] "Generic (PLEG): container finished" podID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerID="25d032ee285c78272d152a9c19fa08bb38e1f5c3ce1ace6eeacc5e57330da4e3" exitCode=0 Nov 01 14:54:13.495713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:13.494981 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" event={"ID":"7d536e10-e370-47e1-a3ed-122c1b806f5b","Type":"ContainerDied","Data":"25d032ee285c78272d152a9c19fa08bb38e1f5c3ce1ace6eeacc5e57330da4e3"} Nov 01 14:54:13.495713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:13.495395 5473 scope.go:122] "RemoveContainer" containerID="25d032ee285c78272d152a9c19fa08bb38e1f5c3ce1ace6eeacc5e57330da4e3" Nov 01 14:54:13.496913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:13.496875 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:13.497048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:13.496946 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:13.497048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:13.496980 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:14.503787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:14.503748 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" event={"ID":"7d536e10-e370-47e1-a3ed-122c1b806f5b","Type":"ContainerStarted","Data":"2585f969f3111591bf5ecb55379254a681ad85ba5103bf7e676eb993b4173472"} Nov 01 14:54:15.291804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:15.291768 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:54:15.292807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:15.292778 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 14:54:15.292956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:15.292871 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 14:54:16.292340 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:16.292306 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 14:54:16.292764 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:16.292359 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 14:54:17.291819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:17.291785 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 14:54:17.292004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:17.291835 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 14:54:17.320905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:17.320876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:17.506254 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:17.506200 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:17.680736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:17.680651 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:18.291910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:18.291875 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 14:54:18.292111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:18.291932 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 14:54:19.291993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:19.291944 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Nov 01 14:54:19.292458 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:19.292029 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Nov 01 14:54:19.785838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:19.785803 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:54:20.278696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.278669 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp"] Nov 01 14:54:20.281804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.281789 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.296372 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.296352 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:54:20.334633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.334607 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp"] Nov 01 14:54:20.336913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.336887 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx96z\" (UniqueName: \"kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.337048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.336940 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.337113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.337093 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.380980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.380951 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:54:20.380980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.380964 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:54:20.438365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.438335 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.438513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.438497 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx96z\" (UniqueName: \"kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.438556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.438521 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.439080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.439059 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.440807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.440783 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.495594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.495566 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx96z\" (UniqueName: \"kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z\") pod \"maas-api-revoke-sub-8d8b148d-q5wgp\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.570329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.570260 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:54:20.594917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.594885 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:20.959284 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:54:20.959256 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc5bbfc_3085_48f7_8230_b3433c507cd3.slice/crio-2876551f06e44f7feaf8c91ce3f560b27a3495cb064e43433287975c3cc50782 WatchSource:0}: Error finding container 2876551f06e44f7feaf8c91ce3f560b27a3495cb064e43433287975c3cc50782: Status 404 returned error can't find the container with id 2876551f06e44f7feaf8c91ce3f560b27a3495cb064e43433287975c3cc50782 Nov 01 14:54:20.963715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:20.963693 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp"] Nov 01 14:54:21.577725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:21.577681 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" event={"ID":"2bc5bbfc-3085-48f7-8230-b3433c507cd3","Type":"ContainerStarted","Data":"013397bffedda06ace2c463f4408b1c948d4fd95d8d0798b8f02d67a8d89a3c0"} Nov 01 14:54:21.578170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:21.577781 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" event={"ID":"2bc5bbfc-3085-48f7-8230-b3433c507cd3","Type":"ContainerStarted","Data":"2876551f06e44f7feaf8c91ce3f560b27a3495cb064e43433287975c3cc50782"} Nov 01 14:54:21.693243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:21.693167 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" podStartSLOduration=1.693154472 podStartE2EDuration="1.693154472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:54:21.690157801 +0000 UTC m=+2200.748309650" watchObservedRunningTime="2026-11-01 14:54:21.693154472 +0000 UTC m=+2200.751306321" Nov 01 14:54:22.098981 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:22.098902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:22.586315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:22.586277 5473 generic.go:356] "Generic (PLEG): container finished" podID="2bc5bbfc-3085-48f7-8230-b3433c507cd3" containerID="013397bffedda06ace2c463f4408b1c948d4fd95d8d0798b8f02d67a8d89a3c0" exitCode=0 Nov 01 14:54:22.586708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:22.586371 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" event={"ID":"2bc5bbfc-3085-48f7-8230-b3433c507cd3","Type":"ContainerDied","Data":"013397bffedda06ace2c463f4408b1c948d4fd95d8d0798b8f02d67a8d89a3c0"} Nov 01 14:54:23.766827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.766799 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:23.884778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.884697 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca\") pod \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " Nov 01 14:54:23.884778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.884760 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca\") pod \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " Nov 01 14:54:23.884993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.884865 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z\" (UniqueName: \"kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z\") pod \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\" (UID: \"2bc5bbfc-3085-48f7-8230-b3433c507cd3\") " Nov 01 14:54:23.886412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.886385 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca" pod "2bc5bbfc-3085-48f7-8230-b3433c507cd3" (UID: "2bc5bbfc-3085-48f7-8230-b3433c507cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:23.888043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.887983 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z" pod "2bc5bbfc-3085-48f7-8230-b3433c507cd3" (UID: "2bc5bbfc-3085-48f7-8230-b3433c507cd3"). InnerVolumeSpecName "kube-api-access-wx96z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:23.888264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.888208 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca" pod "2bc5bbfc-3085-48f7-8230-b3433c507cd3" (UID: "2bc5bbfc-3085-48f7-8230-b3433c507cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:54:23.986598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.986570 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:23.986598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.986594 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc5bbfc-3085-48f7-8230-b3433c507cd3-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:23.986775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:23.986606 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx96z\" (UniqueName: \"kubernetes.io/projected/2bc5bbfc-3085-48f7-8230-b3433c507cd3-kube-api-access-wx96z\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:24.602268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:24.602232 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" Nov 01 14:54:24.602268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:24.602251 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp" event={"ID":"2bc5bbfc-3085-48f7-8230-b3433c507cd3","Type":"ContainerDied","Data":"2876551f06e44f7feaf8c91ce3f560b27a3495cb064e43433287975c3cc50782"} Nov 01 14:54:24.602483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:24.602301 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2876551f06e44f7feaf8c91ce3f560b27a3495cb064e43433287975c3cc50782" Nov 01 14:54:26.504878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:26.504841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:27.516634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.516603 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk"] Nov 01 14:54:27.517108 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.516826 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" containerID="cri-o://738b4ccaa5387b5b42792147808ddc868e785df637ca7396e56047d406947356" gracePeriod=30 Nov 01 14:54:27.567285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.567256 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h"] Nov 01 14:54:27.567499 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.567473 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" containerID="cri-o://2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11" gracePeriod=30 Nov 01 14:54:27.570849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.570825 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6"] Nov 01 14:54:27.571022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.571002 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" containerID="cri-o://d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578" gracePeriod=30 Nov 01 14:54:27.615532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.615440 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-clqh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:54:27.615532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.615484 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:54:27.617642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.617619 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn"] Nov 01 14:54:27.617805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.617784 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" containerID="cri-o://5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0" gracePeriod=30 Nov 01 14:54:27.979905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.979827 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pl9mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:54:27.979905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:27.979882 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:54:31.023529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:31.022945 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:31.107430 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:54:31.107394 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:54:32.272570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.272537 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:32.557631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.557595 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx"] Nov 01 14:54:32.665271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.665237 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb"] Nov 01 14:54:32.697140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.697108 5473 generic.go:356] "Generic (PLEG): container finished" podID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerID="738b4ccaa5387b5b42792147808ddc868e785df637ca7396e56047d406947356" exitCode=0 Nov 01 14:54:32.697466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.697385 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" event={"ID":"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2","Type":"ContainerDied","Data":"738b4ccaa5387b5b42792147808ddc868e785df637ca7396e56047d406947356"} Nov 01 14:54:32.898270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.898250 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907583 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907682 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907713 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907787 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907826 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907911 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907957 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.907997 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.908072 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data\") pod \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\" (UID: \"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2\") " Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.908789 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.909012 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.909575 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.910195 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:32.912362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.910721 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:32.913337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.913244 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "kube-api-access-rkzz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:32.914377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.914355 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:32.914465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.914405 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:54:32.914784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.914764 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token" pod "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" (UID: "3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:32.949194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.949173 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:54:32.970273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:32.962503 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:33.004484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.004459 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:54:33.008541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.008523 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:54:33.009908 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.009891 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.009979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.009952 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010017 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.009984 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010109 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010154 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010168 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.010209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010188 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010233 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010265 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010294 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert\") pod \"b51ee30d-db82-4b0a-afb8-881486f4f982\" (UID: \"b51ee30d-db82-4b0a-afb8-881486f4f982\") " Nov 01 14:54:33.010459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010315 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.010459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010404 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.010459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010425 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.010751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010715 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:33.010841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010819 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010838 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010848 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzz6\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-kube-api-access-rkzz6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010857 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010865 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010873 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010891 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010910 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.010925 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010924 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.011162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010937 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.011162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010950 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.011162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010962 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.011162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010974 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51ee30d-db82-4b0a-afb8-881486f4f982-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.011162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.010986 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.011997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.011964 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.012445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.012425 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "kube-api-access-52qmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:33.012609 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.012592 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:33.012658 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.012592 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo" pod "b51ee30d-db82-4b0a-afb8-881486f4f982" (UID: "b51ee30d-db82-4b0a-afb8-881486f4f982"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:54:33.111704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111678 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.111841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111717 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.111841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111742 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.111963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111863 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.111963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111919 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111961 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.111958 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.112097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112040 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.112314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112104 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112142 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.112314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112157 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.112314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112238 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.112314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112281 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.112314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112311 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112405 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112436 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112434 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112472 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112487 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112509 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112541 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.112649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112641 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data\") pod \"7504900e-d79c-40bf-adfc-fabccbdaf908\" (UID: \"7504900e-d79c-40bf-adfc-fabccbdaf908\") " Nov 01 14:54:33.113081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112656 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.113081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112677 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token\") pod \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\" (UID: \"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec\") " Nov 01 14:54:33.113081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112693 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.113081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112874 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.113081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.112893 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.113630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113605 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.113911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113888 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113919 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113935 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113948 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113960 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113974 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.113990 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114004 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7504900e-d79c-40bf-adfc-fabccbdaf908-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114018 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114033 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52qmq\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-kube-api-access-52qmq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114049 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51ee30d-db82-4b0a-afb8-881486f4f982-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114064 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.114523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114044 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.114857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.114830 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.115181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.115148 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:54:33.115601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.115580 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:33.115741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.115725 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "kube-api-access-f2w2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:33.115818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.115789 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:54:33.115862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.115819 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:54:33.116153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.116129 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token" pod "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" (UID: "73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:33.125538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.125519 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6" pod "7504900e-d79c-40bf-adfc-fabccbdaf908" (UID: "7504900e-d79c-40bf-adfc-fabccbdaf908"). InnerVolumeSpecName "kube-api-access-mr6g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:33.215922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.215745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:33.217183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217147 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217189 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217207 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217240 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217266 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2w2w\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-kube-api-access-f2w2w\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217330 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217347 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7504900e-d79c-40bf-adfc-fabccbdaf908-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217375 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.217477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.217389 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr6g6\" (UniqueName: \"kubernetes.io/projected/7504900e-d79c-40bf-adfc-fabccbdaf908-kube-api-access-mr6g6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:33.707872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.707833 5473 generic.go:356] "Generic (PLEG): container finished" podID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerID="5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0" exitCode=0 Nov 01 14:54:33.708443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.707899 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" Nov 01 14:54:33.708443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.707914 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" event={"ID":"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec","Type":"ContainerDied","Data":"5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0"} Nov 01 14:54:33.708443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.708006 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn" event={"ID":"73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec","Type":"ContainerDied","Data":"6d67ab9ed1733a4edd93fb2083a405f9832e0f87631ff5d0d64fb72f32dc2b81"} Nov 01 14:54:33.708443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.708058 5473 scope.go:122] "RemoveContainer" containerID="5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0" Nov 01 14:54:33.709639 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.709612 5473 generic.go:356] "Generic (PLEG): container finished" podID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerID="d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578" exitCode=0 Nov 01 14:54:33.709774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.709757 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" event={"ID":"7504900e-d79c-40bf-adfc-fabccbdaf908","Type":"ContainerDied","Data":"d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578"} Nov 01 14:54:33.709845 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.709803 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" event={"ID":"7504900e-d79c-40bf-adfc-fabccbdaf908","Type":"ContainerDied","Data":"a183a8e4ecdef528820368b28757cc480bd806ae67972687825e887eddd9d37d"} Nov 01 14:54:33.709901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.709889 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6" Nov 01 14:54:33.718092 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.718067 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" event={"ID":"3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2","Type":"ContainerDied","Data":"25a227f0951e9a14ebe56d60640dfa4382b54a3083a126a78485830ecc11aaca"} Nov 01 14:54:33.718231 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.718071 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk" Nov 01 14:54:33.719547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.719522 5473 generic.go:356] "Generic (PLEG): container finished" podID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerID="2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11" exitCode=0 Nov 01 14:54:33.719643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.719574 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" Nov 01 14:54:33.719643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.719601 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" event={"ID":"b51ee30d-db82-4b0a-afb8-881486f4f982","Type":"ContainerDied","Data":"2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11"} Nov 01 14:54:33.719643 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.719638 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h" event={"ID":"b51ee30d-db82-4b0a-afb8-881486f4f982","Type":"ContainerDied","Data":"b6449683668b3023e0b22d955ca26b8dbb84535e6ec5594b66ef49e10a98b448"} Nov 01 14:54:33.721208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.721185 5473 scope.go:122] "RemoveContainer" containerID="5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0" Nov 01 14:54:33.721537 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:54:33.721518 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0\": container with ID starting with 5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0 not found: ID does not exist" containerID="5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0" Nov 01 14:54:33.721621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.721548 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0"} err="failed to get container status \"5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0\": rpc error: code = NotFound desc = could not find container \"5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0\": container with ID starting with 5ed1729031439c0186768b1b9c03208898b5ac862a3f93574679ca76b1246dd0 not found: ID does not exist" Nov 01 14:54:33.721621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.721573 5473 scope.go:122] "RemoveContainer" containerID="d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578" Nov 01 14:54:33.759850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.759825 5473 scope.go:122] "RemoveContainer" containerID="d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578" Nov 01 14:54:33.760278 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:54:33.760205 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578\": container with ID starting with d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578 not found: ID does not exist" containerID="d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578" Nov 01 14:54:33.760278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.760256 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578"} err="failed to get container status \"d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578\": rpc error: code = NotFound desc = could not find container \"d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578\": container with ID starting with d655cc6e31ee4f6c05d665a718cdd3fba3cda5d1d4fff316e844316e3a02e578 not found: ID does not exist" Nov 01 14:54:33.760398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.760281 5473 scope.go:122] "RemoveContainer" containerID="738b4ccaa5387b5b42792147808ddc868e785df637ca7396e56047d406947356" Nov 01 14:54:33.769823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.769807 5473 scope.go:122] "RemoveContainer" containerID="2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11" Nov 01 14:54:33.779023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.779004 5473 scope.go:122] "RemoveContainer" containerID="2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11" Nov 01 14:54:33.779331 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:54:33.779306 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11\": container with ID starting with 2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11 not found: ID does not exist" containerID="2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11" Nov 01 14:54:33.779398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.779339 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11"} err="failed to get container status \"2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11\": rpc error: code = NotFound desc = could not find container \"2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11\": container with ID starting with 2aafe501e784d7067d83f3687933780854f86d9c3babc57c2ff67d63b0283f11 not found: ID does not exist" Nov 01 14:54:33.873614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.873500 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn"] Nov 01 14:54:33.878922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:33.878893 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dtfjn"] Nov 01 14:54:34.001023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:34.000988 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6"] Nov 01 14:54:34.003836 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:34.003814 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-clqh6"] Nov 01 14:54:34.113445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:34.113411 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h"] Nov 01 14:54:34.116310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:34.116284 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pbj6h"] Nov 01 14:54:34.260291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:34.258271 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk"] Nov 01 14:54:34.269391 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:34.269350 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pl9mk"] Nov 01 14:54:35.082065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:35.082036 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" path="/var/lib/kubelet/pods/3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2/volumes" Nov 01 14:54:35.082528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:35.082514 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" path="/var/lib/kubelet/pods/73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec/volumes" Nov 01 14:54:35.082910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:35.082897 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" path="/var/lib/kubelet/pods/7504900e-d79c-40bf-adfc-fabccbdaf908/volumes" Nov 01 14:54:35.083666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:35.083650 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" path="/var/lib/kubelet/pods/b51ee30d-db82-4b0a-afb8-881486f4f982/volumes" Nov 01 14:54:35.852478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:35.852440 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:37.039298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.039226 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:37.558747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.558707 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" podUID="ce38177d-574e-4ad8-b159-0c0df99fd860" containerName="payload-processing" containerID="cri-o://611d353d39c4660b9a5b6f32bea48ae2847020b4779796c7031d5010be33ce63" gracePeriod=25 Nov 01 14:54:37.665549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.665487 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" podUID="4c193a1b-61c1-45dd-8135-f7c3f13640a9" containerName="payload-pre-processing" containerID="cri-o://60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792" gracePeriod=25 Nov 01 14:54:37.785444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.785410 5473 generic.go:356] "Generic (PLEG): container finished" podID="ce38177d-574e-4ad8-b159-0c0df99fd860" containerID="611d353d39c4660b9a5b6f32bea48ae2847020b4779796c7031d5010be33ce63" exitCode=0 Nov 01 14:54:37.785611 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.785469 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" event={"ID":"ce38177d-574e-4ad8-b159-0c0df99fd860","Type":"ContainerDied","Data":"611d353d39c4660b9a5b6f32bea48ae2847020b4779796c7031d5010be33ce63"} Nov 01 14:54:37.865142 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.864790 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:54:37.892126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.891517 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v\" (UniqueName: \"kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v\") pod \"ce38177d-574e-4ad8-b159-0c0df99fd860\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " Nov 01 14:54:37.892126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.891636 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume\") pod \"ce38177d-574e-4ad8-b159-0c0df99fd860\" (UID: \"ce38177d-574e-4ad8-b159-0c0df99fd860\") " Nov 01 14:54:37.892392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.892326 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume" pod "ce38177d-574e-4ad8-b159-0c0df99fd860" (UID: "ce38177d-574e-4ad8-b159-0c0df99fd860"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:37.895088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.895059 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v" pod "ce38177d-574e-4ad8-b159-0c0df99fd860" (UID: "ce38177d-574e-4ad8-b159-0c0df99fd860"). InnerVolumeSpecName "kube-api-access-d456v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:37.968694 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.968666 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:54:37.992644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.992610 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457\" (UniqueName: \"kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457\") pod \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " Nov 01 14:54:37.992808 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.992775 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume\") pod \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\" (UID: \"4c193a1b-61c1-45dd-8135-f7c3f13640a9\") " Nov 01 14:54:37.993119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.993095 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume" pod "4c193a1b-61c1-45dd-8135-f7c3f13640a9" (UID: "4c193a1b-61c1-45dd-8135-f7c3f13640a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:37.993299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.993283 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c193a1b-61c1-45dd-8135-f7c3f13640a9-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:37.993299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.993300 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d456v\" (UniqueName: \"kubernetes.io/projected/ce38177d-574e-4ad8-b159-0c0df99fd860-kube-api-access-d456v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:37.993467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.993310 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce38177d-574e-4ad8-b159-0c0df99fd860-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:37.995098 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:37.995068 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457" pod "4c193a1b-61c1-45dd-8135-f7c3f13640a9" (UID: "4c193a1b-61c1-45dd-8135-f7c3f13640a9"). InnerVolumeSpecName "kube-api-access-f9457". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:38.094606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.094526 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9457\" (UniqueName: \"kubernetes.io/projected/4c193a1b-61c1-45dd-8135-f7c3f13640a9-kube-api-access-f9457\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:38.281231 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281180 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw"] Nov 01 14:54:38.281728 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281715 5473 cpu_manager.go:414] "containerMap: removing container" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281729 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281743 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281748 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281758 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ce38177d-574e-4ad8-b159-0c0df99fd860" containerName="payload-processing" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281764 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce38177d-574e-4ad8-b159-0c0df99fd860" containerName="payload-processing" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281775 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2bc5bbfc-3085-48f7-8230-b3433c507cd3" containerName="revoke-keys" Nov 01 14:54:38.281782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281780 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc5bbfc-3085-48f7-8230-b3433c507cd3" containerName="revoke-keys" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281794 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281799 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281807 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4c193a1b-61c1-45dd-8135-f7c3f13640a9" containerName="payload-pre-processing" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281811 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c193a1b-61c1-45dd-8135-f7c3f13640a9" containerName="payload-pre-processing" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281822 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281826 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281913 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e6f6f9-8ec9-49c1-9b52-b3fa88a6ebec" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281923 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7504900e-d79c-40bf-adfc-fabccbdaf908" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281929 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce38177d-574e-4ad8-b159-0c0df99fd860" containerName="payload-processing" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281937 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc5bbfc-3085-48f7-8230-b3433c507cd3" containerName="revoke-keys" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281944 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe9ea6d-c5d1-41e2-9620-639f1e19b2f2" containerName="istio-proxy" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281948 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c193a1b-61c1-45dd-8135-f7c3f13640a9" containerName="payload-pre-processing" Nov 01 14:54:38.281999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.281953 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51ee30d-db82-4b0a-afb8-881486f4f982" containerName="istio-proxy" Nov 01 14:54:38.285330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.285311 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.346292 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.342609 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:54:38.357238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.356611 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:54:38.361649 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.361615 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw"] Nov 01 14:54:38.397911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.397881 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.398043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.397930 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.398162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.398138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.499320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.499291 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.499496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.499384 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.499496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.499432 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.500104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.500080 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.501839 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.501815 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.557028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.556995 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj\") pod \"maas-api-revoke-sub-54d3a927-nrlgw\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.598492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.598438 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:38.795378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.795346 5473 generic.go:356] "Generic (PLEG): container finished" podID="4c193a1b-61c1-45dd-8135-f7c3f13640a9" containerID="60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792" exitCode=0 Nov 01 14:54:38.795565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.795410 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" Nov 01 14:54:38.795565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.795429 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" event={"ID":"4c193a1b-61c1-45dd-8135-f7c3f13640a9","Type":"ContainerDied","Data":"60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792"} Nov 01 14:54:38.795565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.795489 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb" event={"ID":"4c193a1b-61c1-45dd-8135-f7c3f13640a9","Type":"ContainerDied","Data":"b858cac8b9c5a580f8de606d63b33b0824805fcd414a3b7f4c1c95ef01276123"} Nov 01 14:54:38.795565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.795507 5473 scope.go:122] "RemoveContainer" containerID="60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792" Nov 01 14:54:38.797026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.797004 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" event={"ID":"ce38177d-574e-4ad8-b159-0c0df99fd860","Type":"ContainerDied","Data":"260d457b0a4e3dbc4c9176543f331e68aaa233c6bc5867d4e4b3739c11772b18"} Nov 01 14:54:38.797143 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.797029 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx" Nov 01 14:54:38.806576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.806522 5473 scope.go:122] "RemoveContainer" containerID="60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792" Nov 01 14:54:38.806911 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:54:38.806889 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792\": container with ID starting with 60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792 not found: ID does not exist" containerID="60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792" Nov 01 14:54:38.806988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.806918 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792"} err="failed to get container status \"60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792\": rpc error: code = NotFound desc = could not find container \"60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792\": container with ID starting with 60abfa35d1986326d9eab3fa631fb950b28726219d4b86ea0fa96f36cd436792 not found: ID does not exist" Nov 01 14:54:38.806988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.806934 5473 scope.go:122] "RemoveContainer" containerID="611d353d39c4660b9a5b6f32bea48ae2847020b4779796c7031d5010be33ce63" Nov 01 14:54:38.961901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.961869 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb"] Nov 01 14:54:38.965202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:38.965177 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-288f1e-7bc4989d7-98rtb"] Nov 01 14:54:38.993076 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:54:38.993048 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a53bcf_a11f_4f27_8a0d_4564ab53746d.slice/crio-1d8b009049511d2170157b29348c7e11c21429b37045e5e9e7bc8a3bb6d457f6 WatchSource:0}: Error finding container 1d8b009049511d2170157b29348c7e11c21429b37045e5e9e7bc8a3bb6d457f6: Status 404 returned error can't find the container with id 1d8b009049511d2170157b29348c7e11c21429b37045e5e9e7bc8a3bb6d457f6 Nov 01 14:54:39.013945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.013920 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw"] Nov 01 14:54:39.084520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.084491 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c193a1b-61c1-45dd-8135-f7c3f13640a9" path="/var/lib/kubelet/pods/4c193a1b-61c1-45dd-8135-f7c3f13640a9/volumes" Nov 01 14:54:39.084810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.084795 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx"] Nov 01 14:54:39.084882 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.084824 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-288f1e-6664cfb79f-jl5mx"] Nov 01 14:54:39.808918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.808872 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" event={"ID":"e4a53bcf-a11f-4f27-8a0d-4564ab53746d","Type":"ContainerStarted","Data":"2a3314c6544228f6c232c798f20aac52d87421a5596e27228e4780998978ec4d"} Nov 01 14:54:39.809481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.808937 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" event={"ID":"e4a53bcf-a11f-4f27-8a0d-4564ab53746d","Type":"ContainerStarted","Data":"1d8b009049511d2170157b29348c7e11c21429b37045e5e9e7bc8a3bb6d457f6"} Nov 01 14:54:39.923615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:39.923568 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" podStartSLOduration=1.9235555 podStartE2EDuration="1.9235555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:54:39.922794771 +0000 UTC m=+2218.980946620" watchObservedRunningTime="2026-11-01 14:54:39.9235555 +0000 UTC m=+2218.981707348" Nov 01 14:54:40.307800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:40.307731 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:40.818239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:40.818180 5473 generic.go:356] "Generic (PLEG): container finished" podID="e4a53bcf-a11f-4f27-8a0d-4564ab53746d" containerID="2a3314c6544228f6c232c798f20aac52d87421a5596e27228e4780998978ec4d" exitCode=0 Nov 01 14:54:40.818599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:40.818258 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" event={"ID":"e4a53bcf-a11f-4f27-8a0d-4564ab53746d","Type":"ContainerDied","Data":"2a3314c6544228f6c232c798f20aac52d87421a5596e27228e4780998978ec4d"} Nov 01 14:54:41.090588 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:41.090510 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce38177d-574e-4ad8-b159-0c0df99fd860" path="/var/lib/kubelet/pods/ce38177d-574e-4ad8-b159-0c0df99fd860/volumes" Nov 01 14:54:41.973704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:41.973679 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:42.037320 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.037286 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj\") pod \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " Nov 01 14:54:42.037496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.037335 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca\") pod \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " Nov 01 14:54:42.037568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.037506 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca\") pod \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\" (UID: \"e4a53bcf-a11f-4f27-8a0d-4564ab53746d\") " Nov 01 14:54:42.037832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.037809 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca" pod "e4a53bcf-a11f-4f27-8a0d-4564ab53746d" (UID: "e4a53bcf-a11f-4f27-8a0d-4564ab53746d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:42.037940 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.037900 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:42.039850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.039825 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca" pod "e4a53bcf-a11f-4f27-8a0d-4564ab53746d" (UID: "e4a53bcf-a11f-4f27-8a0d-4564ab53746d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:54:42.039956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.039928 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj" pod "e4a53bcf-a11f-4f27-8a0d-4564ab53746d" (UID: "e4a53bcf-a11f-4f27-8a0d-4564ab53746d"). InnerVolumeSpecName "kube-api-access-bgtgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:42.138553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.138485 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgtgj\" (UniqueName: \"kubernetes.io/projected/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-kube-api-access-bgtgj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:42.138553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.138514 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4a53bcf-a11f-4f27-8a0d-4564ab53746d-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:42.725781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.725748 5473 scope.go:122] "RemoveContainer" containerID="1e40cf29b994f0a638a0a21fb9d6d71485f4a970f7e7566786d466c6ca68a552" Nov 01 14:54:42.834382 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.834359 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" Nov 01 14:54:42.834531 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.834387 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw" event={"ID":"e4a53bcf-a11f-4f27-8a0d-4564ab53746d","Type":"ContainerDied","Data":"1d8b009049511d2170157b29348c7e11c21429b37045e5e9e7bc8a3bb6d457f6"} Nov 01 14:54:42.834531 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.834436 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d8b009049511d2170157b29348c7e11c21429b37045e5e9e7bc8a3bb6d457f6" Nov 01 14:54:42.835914 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.835894 5473 generic.go:356] "Generic (PLEG): container finished" podID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerID="4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1" exitCode=0 Nov 01 14:54:42.836046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.835962 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" event={"ID":"ff0c815e-4363-4ce1-ad46-bccbe276941b","Type":"ContainerDied","Data":"4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1"} Nov 01 14:54:42.836385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.836368 5473 scope.go:122] "RemoveContainer" containerID="4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1" Nov 01 14:54:42.838009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.837978 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:42.838082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.838064 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:42.838145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.838131 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:42.974002 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:42.973976 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:54:43.855520 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:43.855482 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" event={"ID":"ff0c815e-4363-4ce1-ad46-bccbe276941b","Type":"ContainerStarted","Data":"0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407"} Nov 01 14:54:44.750528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:44.750492 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:44.789873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:44.789839 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:54:44.791023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:44.790998 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 01 14:54:44.791155 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:44.791047 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 01 14:54:45.790120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:45.790074 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 01 14:54:45.790622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:45.790147 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 01 14:54:46.789828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:46.789794 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 01 14:54:46.790008 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:46.789855 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 01 14:54:47.790018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:47.789980 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Nov 01 14:54:47.790419 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:47.790050 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Nov 01 14:54:48.817249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:48.816714 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:54:48.817718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:48.817614 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:54:48.818523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:48.818499 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:54:49.172599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:49.172522 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:50.234518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.234482 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m"] Nov 01 14:54:50.235248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.235196 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e4a53bcf-a11f-4f27-8a0d-4564ab53746d" containerName="revoke-keys" Nov 01 14:54:50.235248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.235233 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a53bcf-a11f-4f27-8a0d-4564ab53746d" containerName="revoke-keys" Nov 01 14:54:50.235433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.235419 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a53bcf-a11f-4f27-8a0d-4564ab53746d" containerName="revoke-keys" Nov 01 14:54:50.239494 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.239473 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.292356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.292335 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-536a87-openshift-default-dockercfg-588rv\"" Nov 01 14:54:50.329642 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.329616 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.329761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.329658 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.329830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.329807 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.329890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.329868 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.329972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.329945 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.330153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.330134 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7c6f\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.330288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.330167 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.330288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.330206 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.330403 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.330379 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.347709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.347685 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m"] Nov 01 14:54:50.431902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.431876 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.431907 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.431933 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.431963 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.431993 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432069 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7c6f\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432102 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432139 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432205 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432363 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432442 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432470 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432630 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.432961 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.432942 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.434255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.434233 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.434655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.434638 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.489430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.489364 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.489576 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.489553 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7c6f\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.553395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.553361 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:50.972267 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:54:50.972239 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef256c70_9de7_4d17_b204_6239b96e14b3.slice/crio-dcff0908c87f4ee95996107804ebd01a7f9f4e2a599b1ab1e71a26187780e5f8 WatchSource:0}: Error finding container dcff0908c87f4ee95996107804ebd01a7f9f4e2a599b1ab1e71a26187780e5f8: Status 404 returned error can't find the container with id dcff0908c87f4ee95996107804ebd01a7f9f4e2a599b1ab1e71a26187780e5f8 Nov 01 14:54:50.974991 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.974957 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:50.975100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.975034 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:50.975100 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.975076 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:54:50.975585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:50.975531 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m"] Nov 01 14:54:51.911039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:51.911000 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" event={"ID":"ef256c70-9de7-4d17-b204-6239b96e14b3","Type":"ContainerStarted","Data":"4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4"} Nov 01 14:54:51.911465 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:51.911053 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" event={"ID":"ef256c70-9de7-4d17-b204-6239b96e14b3","Type":"ContainerStarted","Data":"dcff0908c87f4ee95996107804ebd01a7f9f4e2a599b1ab1e71a26187780e5f8"} Nov 01 14:54:52.140023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:52.139940 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podStartSLOduration=2.139920887 podStartE2EDuration="2.139920887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:54:52.077847948 +0000 UTC m=+2231.135999797" watchObservedRunningTime="2026-11-01 14:54:52.139920887 +0000 UTC m=+2231.198072734" Nov 01 14:54:52.554314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:52.553942 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:52.555693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:52.555653 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 14:54:52.555867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:52.555735 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 14:54:53.553897 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:53.553866 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 14:54:53.554387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:53.553932 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 14:54:53.715810 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:53.715774 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:54.294280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.294251 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22"] Nov 01 14:54:54.298679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.298657 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.354014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.353982 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22"] Nov 01 14:54:54.382495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.382467 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.382648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.382518 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.382729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.382702 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvbc\" (UniqueName: \"kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.399673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.399654 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:54:54.399788 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.399772 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:54:54.483915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.483884 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.484118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.483926 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.484118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.484001 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvbc\" (UniqueName: \"kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.484631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.484609 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.486358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.486337 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.550751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.550680 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvbc\" (UniqueName: \"kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc\") pod \"maas-api-revoke-sub-eae65426-jkm22\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.554526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.554502 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 14:54:54.554895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.554549 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 14:54:54.613666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.613642 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:54.795385 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:54.795353 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:55.008191 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:54:55.008164 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1a3d2b_7856_4e02_bee1_b89ac1660229.slice/crio-c46c85c16c27d045aa580ae52453bc7c1f461b123bc57a9a92a438c8d4305dfd WatchSource:0}: Error finding container c46c85c16c27d045aa580ae52453bc7c1f461b123bc57a9a92a438c8d4305dfd: Status 404 returned error can't find the container with id c46c85c16c27d045aa580ae52453bc7c1f461b123bc57a9a92a438c8d4305dfd Nov 01 14:54:55.011350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:55.011327 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22"] Nov 01 14:54:55.444785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:55.444702 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:55.554044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:55.554007 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 14:54:55.554248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:55.554071 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 14:54:55.964562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:55.964442 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" event={"ID":"db1a3d2b-7856-4e02-bee1-b89ac1660229","Type":"ContainerStarted","Data":"584b3a84b73cd60f47fa7e165cd81a2b4e17769ae7376328cc37582e679758d9"} Nov 01 14:54:55.964562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:55.964506 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" event={"ID":"db1a3d2b-7856-4e02-bee1-b89ac1660229","Type":"ContainerStarted","Data":"c46c85c16c27d045aa580ae52453bc7c1f461b123bc57a9a92a438c8d4305dfd"} Nov 01 14:54:56.085482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:56.085433 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" podStartSLOduration=2.085418329 podStartE2EDuration="2.085418329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:54:56.084692782 +0000 UTC m=+2235.142844632" watchObservedRunningTime="2026-11-01 14:54:56.085418329 +0000 UTC m=+2235.143570179" Nov 01 14:54:56.553784 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:56.553754 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 14:54:56.553969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:56.553802 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 14:54:56.972314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:56.972201 5473 generic.go:356] "Generic (PLEG): container finished" podID="db1a3d2b-7856-4e02-bee1-b89ac1660229" containerID="584b3a84b73cd60f47fa7e165cd81a2b4e17769ae7376328cc37582e679758d9" exitCode=0 Nov 01 14:54:56.972314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:56.972284 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" event={"ID":"db1a3d2b-7856-4e02-bee1-b89ac1660229","Type":"ContainerDied","Data":"584b3a84b73cd60f47fa7e165cd81a2b4e17769ae7376328cc37582e679758d9"} Nov 01 14:54:57.554402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:57.554364 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 01 14:54:57.554604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:57.554432 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 01 14:54:58.148276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.148251 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:58.165987 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.165956 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:58.230163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.230130 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca\") pod \"db1a3d2b-7856-4e02-bee1-b89ac1660229\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " Nov 01 14:54:58.230347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.230268 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca\") pod \"db1a3d2b-7856-4e02-bee1-b89ac1660229\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " Nov 01 14:54:58.230467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.230449 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc\" (UniqueName: \"kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc\") pod \"db1a3d2b-7856-4e02-bee1-b89ac1660229\" (UID: \"db1a3d2b-7856-4e02-bee1-b89ac1660229\") " Nov 01 14:54:58.230567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.230512 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca" pod "db1a3d2b-7856-4e02-bee1-b89ac1660229" (UID: "db1a3d2b-7856-4e02-bee1-b89ac1660229"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:54:58.230903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.230877 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:58.232417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.232390 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca" pod "db1a3d2b-7856-4e02-bee1-b89ac1660229" (UID: "db1a3d2b-7856-4e02-bee1-b89ac1660229"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:54:58.232530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.232485 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc" pod "db1a3d2b-7856-4e02-bee1-b89ac1660229" (UID: "db1a3d2b-7856-4e02-bee1-b89ac1660229"). InnerVolumeSpecName "kube-api-access-kgvbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:54:58.332172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.332122 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvbc\" (UniqueName: \"kubernetes.io/projected/db1a3d2b-7856-4e02-bee1-b89ac1660229-kube-api-access-kgvbc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:58.332172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.332155 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db1a3d2b-7856-4e02-bee1-b89ac1660229-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:54:58.394383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.394348 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:54:58.558654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.558627 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:58.559039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.559015 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:58.559598 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.559579 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 14:54:58.988107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.988028 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" Nov 01 14:54:58.988107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.988063 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22" event={"ID":"db1a3d2b-7856-4e02-bee1-b89ac1660229","Type":"ContainerDied","Data":"c46c85c16c27d045aa580ae52453bc7c1f461b123bc57a9a92a438c8d4305dfd"} Nov 01 14:54:58.988393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:58.988114 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c46c85c16c27d045aa580ae52453bc7c1f461b123bc57a9a92a438c8d4305dfd" Nov 01 14:54:59.151766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:54:59.151736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:00.255065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.255032 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:00.772920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.772882 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm"] Nov 01 14:55:00.773478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.773463 5473 cpu_manager.go:414] "containerMap: removing container" podUID="db1a3d2b-7856-4e02-bee1-b89ac1660229" containerName="revoke-keys" Nov 01 14:55:00.773549 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.773480 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1a3d2b-7856-4e02-bee1-b89ac1660229" containerName="revoke-keys" Nov 01 14:55:00.773592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.773578 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1a3d2b-7856-4e02-bee1-b89ac1660229" containerName="revoke-keys" Nov 01 14:55:00.777115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.777099 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.851252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.851188 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-7197e2-openshift-default-dockercfg-s6tb5\"" Nov 01 14:55:00.864510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.864457 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.867950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.867619 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.867950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.867865 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.868139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.868051 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29lrm\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.868278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.868199 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.868377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.868310 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.868377 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.868341 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.868503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.868388 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.868503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.868413 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.893132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.893100 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:00.905107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.905082 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm"] Nov 01 14:55:00.970088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970053 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970107 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970130 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970157 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970174 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970259 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970292 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970314 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970352 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29lrm\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970719 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970767 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970890 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970889 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.970949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.970915 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.973051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.973030 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:00.973203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:00.973187 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:01.029369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.029306 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:01.029526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.029508 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29lrm\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm\") pod \"e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:01.090768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.090732 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:01.116632 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:55:01.116590 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:55:01.137281 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.137251 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:01.516823 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:01.516794 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc9c369_f74e_4f5a_b5e6_ae238c5cba32.slice/crio-506f85f9fbc158fca00bf24ebdd9f5f74e63bce522621ad57c47012b1cdea724 WatchSource:0}: Error finding container 506f85f9fbc158fca00bf24ebdd9f5f74e63bce522621ad57c47012b1cdea724: Status 404 returned error can't find the container with id 506f85f9fbc158fca00bf24ebdd9f5f74e63bce522621ad57c47012b1cdea724 Nov 01 14:55:01.519513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.519478 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:55:01.519603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.519548 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:55:01.519603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.519577 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:55:01.519816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:01.519794 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm"] Nov 01 14:55:02.010645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:02.010560 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" event={"ID":"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32","Type":"ContainerStarted","Data":"ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d"} Nov 01 14:55:02.010645 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:02.010615 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" event={"ID":"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32","Type":"ContainerStarted","Data":"506f85f9fbc158fca00bf24ebdd9f5f74e63bce522621ad57c47012b1cdea724"} Nov 01 14:55:02.091735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:02.091703 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:02.097636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:02.097611 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:02.128903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:02.128843 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podStartSLOduration=2.128803503 podStartE2EDuration="2.128803503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:02.126991323 +0000 UTC m=+2241.185143173" watchObservedRunningTime="2026-11-01 14:55:02.128803503 +0000 UTC m=+2241.186955352" Nov 01 14:55:03.017613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:03.017579 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:03.018496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:03.018480 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:55:03.213424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:03.213386 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:04.193774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:04.193741 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:04.861322 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:04.861283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:05.054192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:05.054153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:05.227818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:05.227728 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:08.097720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:08.097664 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:10.106110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.106078 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:10.119702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.119673 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4"] Nov 01 14:55:10.123711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.123695 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.176287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.176246 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:55:10.203140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.203105 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvvn\" (UniqueName: \"kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.203311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.203249 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.203389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.203367 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.226305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.226270 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:55:10.229526 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.229502 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4"] Nov 01 14:55:10.305112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.305081 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvvn\" (UniqueName: \"kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.305342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.305163 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.305342 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.305288 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.305924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.305905 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.307889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.307867 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.362178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.362108 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvvn\" (UniqueName: \"kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn\") pod \"maas-api-revoke-sub-90489d23-vq5b4\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.406640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.406593 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:10.438389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.438360 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:10.573312 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.570966 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:10.780709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.780676 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk"] Nov 01 14:55:10.785015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.784999 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:10.847338 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.846642 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7197e2\"" Nov 01 14:55:10.908380 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.908283 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7197e2-dockercfg-gcbc9\"" Nov 01 14:55:10.919639 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.919611 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk"] Nov 01 14:55:10.923738 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.923700 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4"] Nov 01 14:55:10.927492 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:10.927417 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce37353_56fc_4a07_bdc4_7a41e8c28f1a.slice/crio-e9fd19fb036b4f86a0e2b54ba3afa7080f0d85d061af648a5a614eeca2d7e526 WatchSource:0}: Error finding container e9fd19fb036b4f86a0e2b54ba3afa7080f0d85d061af648a5a614eeca2d7e526: Status 404 returned error can't find the container with id e9fd19fb036b4f86a0e2b54ba3afa7080f0d85d061af648a5a614eeca2d7e526 Nov 01 14:55:10.947623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.947596 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74"] Nov 01 14:55:10.953341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.953319 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:10.977412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.977388 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7244g\" (UniqueName: \"kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g\") pod \"payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:10.977564 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:10.977548 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:11.004242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.004198 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74"] Nov 01 14:55:11.078727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.078643 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7244g\" (UniqueName: \"kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g\") pod \"payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:11.078873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.078815 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggkb6\" (UniqueName: \"kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6\") pod \"payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.078873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.078855 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:11.079006 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.078972 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.079664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.079643 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:11.084026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.084003 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" event={"ID":"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a","Type":"ContainerStarted","Data":"e9fd19fb036b4f86a0e2b54ba3afa7080f0d85d061af648a5a614eeca2d7e526"} Nov 01 14:55:11.135496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.135474 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7244g\" (UniqueName: \"kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g\") pod \"payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:11.180640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.180609 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggkb6\" (UniqueName: \"kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6\") pod \"payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.180775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.180682 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.181443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.181421 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.237211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.237182 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggkb6\" (UniqueName: \"kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6\") pod \"payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.287968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.287940 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:11.400865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.400821 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:11.654929 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:11.654889 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1f9291_f31d_4a6c_9bab_25004dd21fee.slice/crio-d326f50aca05696670cf591c9b6b4a5ae3702b14627d51c7f972fa8a6f6b6e37 WatchSource:0}: Error finding container d326f50aca05696670cf591c9b6b4a5ae3702b14627d51c7f972fa8a6f6b6e37: Status 404 returned error can't find the container with id d326f50aca05696670cf591c9b6b4a5ae3702b14627d51c7f972fa8a6f6b6e37 Nov 01 14:55:11.658592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.658563 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74"] Nov 01 14:55:11.769835 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:11.769810 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ffa5cc_e50d_4829_97b0_85c0e826ecef.slice/crio-f9a0f217c52d59d56c85dcd91ffabf6299558728a473f6923086122709417f60 WatchSource:0}: Error finding container f9a0f217c52d59d56c85dcd91ffabf6299558728a473f6923086122709417f60: Status 404 returned error can't find the container with id f9a0f217c52d59d56c85dcd91ffabf6299558728a473f6923086122709417f60 Nov 01 14:55:11.773453 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:11.773425 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk"] Nov 01 14:55:12.092682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.092647 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" event={"ID":"e9ffa5cc-e50d-4829-97b0-85c0e826ecef","Type":"ContainerStarted","Data":"341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486"} Nov 01 14:55:12.092847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.092711 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:12.092847 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.092732 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" event={"ID":"e9ffa5cc-e50d-4829-97b0-85c0e826ecef","Type":"ContainerStarted","Data":"f9a0f217c52d59d56c85dcd91ffabf6299558728a473f6923086122709417f60"} Nov 01 14:55:12.094287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.094261 5473 generic.go:356] "Generic (PLEG): container finished" podID="9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" containerID="1e3112b612a06493059c192e09e9257ffe70fbafd64d60e4ffe4f544771bfee6" exitCode=0 Nov 01 14:55:12.094389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.094331 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" event={"ID":"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a","Type":"ContainerDied","Data":"1e3112b612a06493059c192e09e9257ffe70fbafd64d60e4ffe4f544771bfee6"} Nov 01 14:55:12.095753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.095727 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" event={"ID":"ef1f9291-f31d-4a6c-9bab-25004dd21fee","Type":"ContainerStarted","Data":"95e227edf2f01087a14c46c61ed42d1daa6b61b4c393130c4842ccb607feb51e"} Nov 01 14:55:12.095834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.095772 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" event={"ID":"ef1f9291-f31d-4a6c-9bab-25004dd21fee","Type":"ContainerStarted","Data":"d326f50aca05696670cf591c9b6b4a5ae3702b14627d51c7f972fa8a6f6b6e37"} Nov 01 14:55:12.095834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.095795 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:12.208262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.208190 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" podStartSLOduration=2.208176461 podStartE2EDuration="2.208176461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:12.206425872 +0000 UTC m=+2251.264577723" watchObservedRunningTime="2026-11-01 14:55:12.208176461 +0000 UTC m=+2251.266328311" Nov 01 14:55:12.444140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.443887 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" podStartSLOduration=2.443865602 podStartE2EDuration="2.443865602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:12.326430498 +0000 UTC m=+2251.384582349" watchObservedRunningTime="2026-11-01 14:55:12.443865602 +0000 UTC m=+2251.502017453" Nov 01 14:55:12.695949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:12.695861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:13.257751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.257725 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:13.407236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.407131 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca\") pod \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " Nov 01 14:55:13.407236 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.407201 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca\") pod \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " Nov 01 14:55:13.407446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.407405 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn\" (UniqueName: \"kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn\") pod \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\" (UID: \"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a\") " Nov 01 14:55:13.407565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.407542 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca" pod "9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" (UID: "9ce37353-56fc-4a07-bdc4-7a41e8c28f1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:55:13.407899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.407875 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:13.409263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.409242 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca" pod "9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" (UID: "9ce37353-56fc-4a07-bdc4-7a41e8c28f1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:55:13.409404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.409384 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn" pod "9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" (UID: "9ce37353-56fc-4a07-bdc4-7a41e8c28f1a"). InnerVolumeSpecName "kube-api-access-stvvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:55:13.508659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.508628 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvvn\" (UniqueName: \"kubernetes.io/projected/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-kube-api-access-stvvn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:13.508659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:13.508656 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:14.110045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:14.110010 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" event={"ID":"9ce37353-56fc-4a07-bdc4-7a41e8c28f1a","Type":"ContainerDied","Data":"e9fd19fb036b4f86a0e2b54ba3afa7080f0d85d061af648a5a614eeca2d7e526"} Nov 01 14:55:14.110045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:14.110036 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4" Nov 01 14:55:14.110270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:14.110065 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9fd19fb036b4f86a0e2b54ba3afa7080f0d85d061af648a5a614eeca2d7e526" Nov 01 14:55:18.717985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:18.717951 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:23.097196 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:23.097119 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:23.104884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:23.104857 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:55:23.105062 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:23.105019 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:55:26.036551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.036510 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb"] Nov 01 14:55:26.037328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.037305 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" containerName="revoke-keys" Nov 01 14:55:26.037328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.037327 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" containerName="revoke-keys" Nov 01 14:55:26.037506 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.037460 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" containerName="revoke-keys" Nov 01 14:55:26.042144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.042123 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.092805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.092776 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb"] Nov 01 14:55:26.138975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.138941 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:55:26.139129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.139009 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:55:26.189230 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.189184 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.189392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.189327 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47v44\" (UniqueName: \"kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.189532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.189509 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.290755 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.290661 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47v44\" (UniqueName: \"kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.290930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.290808 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.290930 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.290924 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.291529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.291509 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.293256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.293239 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.347843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.347814 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47v44\" (UniqueName: \"kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44\") pod \"maas-api-revoke-sub-1246786d-m5fwb\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.356101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.356079 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:26.723087 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:26.723059 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81b8cbc_02ef_41e6_8717_d036c01b7eed.slice/crio-c165d50b737ba256ce5499d8473028b0d3e052493d57a83524afa5a156007bb0 WatchSource:0}: Error finding container c165d50b737ba256ce5499d8473028b0d3e052493d57a83524afa5a156007bb0: Status 404 returned error can't find the container with id c165d50b737ba256ce5499d8473028b0d3e052493d57a83524afa5a156007bb0 Nov 01 14:55:26.728248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:26.728209 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb"] Nov 01 14:55:27.199538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:27.199497 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" event={"ID":"d81b8cbc-02ef-41e6-8717-d036c01b7eed","Type":"ContainerStarted","Data":"8e406613191e659f8929f4a9a2de7687afe5ffaf044167853c30d70ff3390eb4"} Nov 01 14:55:27.199943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:27.199552 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" event={"ID":"d81b8cbc-02ef-41e6-8717-d036c01b7eed","Type":"ContainerStarted","Data":"c165d50b737ba256ce5499d8473028b0d3e052493d57a83524afa5a156007bb0"} Nov 01 14:55:27.316137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:27.316018 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" podStartSLOduration=2.316003356 podStartE2EDuration="2.316003356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:27.312358959 +0000 UTC m=+2266.370510800" watchObservedRunningTime="2026-11-01 14:55:27.316003356 +0000 UTC m=+2266.374155249" Nov 01 14:55:27.450344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:27.450312 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:28.169133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:28.169098 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:28.209466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:28.209436 5473 generic.go:356] "Generic (PLEG): container finished" podID="d81b8cbc-02ef-41e6-8717-d036c01b7eed" containerID="8e406613191e659f8929f4a9a2de7687afe5ffaf044167853c30d70ff3390eb4" exitCode=0 Nov 01 14:55:28.209870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:28.209516 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" event={"ID":"d81b8cbc-02ef-41e6-8717-d036c01b7eed","Type":"ContainerDied","Data":"8e406613191e659f8929f4a9a2de7687afe5ffaf044167853c30d70ff3390eb4"} Nov 01 14:55:29.430104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.430077 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:29.529843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.529801 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44\" (UniqueName: \"kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44\") pod \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " Nov 01 14:55:29.530050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.529866 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca\") pod \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " Nov 01 14:55:29.530050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.530022 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca\") pod \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\" (UID: \"d81b8cbc-02ef-41e6-8717-d036c01b7eed\") " Nov 01 14:55:29.530684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.530657 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca" pod "d81b8cbc-02ef-41e6-8717-d036c01b7eed" (UID: "d81b8cbc-02ef-41e6-8717-d036c01b7eed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:55:29.532436 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.532409 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca" pod "d81b8cbc-02ef-41e6-8717-d036c01b7eed" (UID: "d81b8cbc-02ef-41e6-8717-d036c01b7eed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:55:29.532636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.532614 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44" pod "d81b8cbc-02ef-41e6-8717-d036c01b7eed" (UID: "d81b8cbc-02ef-41e6-8717-d036c01b7eed"). InnerVolumeSpecName "kube-api-access-47v44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:55:29.631825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.631787 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:29.631825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.631822 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d81b8cbc-02ef-41e6-8717-d036c01b7eed-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:29.632112 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:29.631835 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47v44\" (UniqueName: \"kubernetes.io/projected/d81b8cbc-02ef-41e6-8717-d036c01b7eed-kube-api-access-47v44\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:30.226913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:30.226881 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" Nov 01 14:55:30.227102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:30.226920 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb" event={"ID":"d81b8cbc-02ef-41e6-8717-d036c01b7eed","Type":"ContainerDied","Data":"c165d50b737ba256ce5499d8473028b0d3e052493d57a83524afa5a156007bb0"} Nov 01 14:55:30.227102 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:30.226976 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c165d50b737ba256ce5499d8473028b0d3e052493d57a83524afa5a156007bb0" Nov 01 14:55:31.107771 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:55:31.107740 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:55:31.409740 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:31.409654 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:31.874383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:31.874330 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:34.258566 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:34.258534 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:35.091890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:35.091859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:35.249614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:35.249581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:36.720781 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.720752 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p"] Nov 01 14:55:36.721421 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.721404 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d81b8cbc-02ef-41e6-8717-d036c01b7eed" containerName="revoke-keys" Nov 01 14:55:36.721482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.721422 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81b8cbc-02ef-41e6-8717-d036c01b7eed" containerName="revoke-keys" Nov 01 14:55:36.721528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.721515 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81b8cbc-02ef-41e6-8717-d036c01b7eed" containerName="revoke-keys" Nov 01 14:55:36.725467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.725447 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.778110 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.778082 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p"] Nov 01 14:55:36.822652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.822628 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:55:36.822790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.822684 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:55:36.824243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.824201 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.824368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.824350 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.824426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.824415 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgth\" (UniqueName: \"kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.925702 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.925669 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.925887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.925796 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.925887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.925843 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgth\" (UniqueName: \"kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.926420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.926399 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.928080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.928057 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:36.982975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:36.982903 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgth\" (UniqueName: \"kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth\") pod \"maas-api-revoke-sub-eda9c242-4pd9p\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:37.038721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:37.038690 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:37.406058 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:37.406027 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26179a37_0712_40ec_8d08_a824aa6fb6a2.slice/crio-0f4836a0588e03740d281ddf523e3db72edbc8197686ef5e683c6aaaae0534c0 WatchSource:0}: Error finding container 0f4836a0588e03740d281ddf523e3db72edbc8197686ef5e683c6aaaae0534c0: Status 404 returned error can't find the container with id 0f4836a0588e03740d281ddf523e3db72edbc8197686ef5e683c6aaaae0534c0 Nov 01 14:55:37.409451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:37.409428 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p"] Nov 01 14:55:38.293543 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:38.293510 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" event={"ID":"26179a37-0712-40ec-8d08-a824aa6fb6a2","Type":"ContainerStarted","Data":"546f583c792d50a37e197c72739ccaccff8d8b89c15793507e246cd30ecbb18e"} Nov 01 14:55:38.294024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:38.293562 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" event={"ID":"26179a37-0712-40ec-8d08-a824aa6fb6a2","Type":"ContainerStarted","Data":"0f4836a0588e03740d281ddf523e3db72edbc8197686ef5e683c6aaaae0534c0"} Nov 01 14:55:38.407532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:38.407485 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" podStartSLOduration=2.407472289 podStartE2EDuration="2.407472289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:55:38.406821491 +0000 UTC m=+2277.464973340" watchObservedRunningTime="2026-11-01 14:55:38.407472289 +0000 UTC m=+2277.465624138" Nov 01 14:55:38.531592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:38.531550 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:38.706636 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:38.706546 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:39.301720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:39.301686 5473 generic.go:356] "Generic (PLEG): container finished" podID="26179a37-0712-40ec-8d08-a824aa6fb6a2" containerID="546f583c792d50a37e197c72739ccaccff8d8b89c15793507e246cd30ecbb18e" exitCode=0 Nov 01 14:55:39.302179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:39.301733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" event={"ID":"26179a37-0712-40ec-8d08-a824aa6fb6a2","Type":"ContainerDied","Data":"546f583c792d50a37e197c72739ccaccff8d8b89c15793507e246cd30ecbb18e"} Nov 01 14:55:40.463134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.463108 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:40.472860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.472839 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth\" (UniqueName: \"kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth\") pod \"26179a37-0712-40ec-8d08-a824aa6fb6a2\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " Nov 01 14:55:40.472997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.472906 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca\") pod \"26179a37-0712-40ec-8d08-a824aa6fb6a2\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " Nov 01 14:55:40.473061 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.473025 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca\") pod \"26179a37-0712-40ec-8d08-a824aa6fb6a2\" (UID: \"26179a37-0712-40ec-8d08-a824aa6fb6a2\") " Nov 01 14:55:40.473235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.473201 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca" pod "26179a37-0712-40ec-8d08-a824aa6fb6a2" (UID: "26179a37-0712-40ec-8d08-a824aa6fb6a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:55:40.473406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.473391 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:40.474887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.474859 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca" pod "26179a37-0712-40ec-8d08-a824aa6fb6a2" (UID: "26179a37-0712-40ec-8d08-a824aa6fb6a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:55:40.474990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.474881 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth" pod "26179a37-0712-40ec-8d08-a824aa6fb6a2" (UID: "26179a37-0712-40ec-8d08-a824aa6fb6a2"). InnerVolumeSpecName "kube-api-access-mxgth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:55:40.575049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.574971 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26179a37-0712-40ec-8d08-a824aa6fb6a2-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:40.575049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:40.575005 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxgth\" (UniqueName: \"kubernetes.io/projected/26179a37-0712-40ec-8d08-a824aa6fb6a2-kube-api-access-mxgth\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:41.317130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:41.317100 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 14:55:41.317323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:41.317134 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" event={"ID":"26179a37-0712-40ec-8d08-a824aa6fb6a2","Type":"ContainerDied","Data":"0f4836a0588e03740d281ddf523e3db72edbc8197686ef5e683c6aaaae0534c0"} Nov 01 14:55:41.317323 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:41.317176 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4836a0588e03740d281ddf523e3db72edbc8197686ef5e683c6aaaae0534c0" Nov 01 14:55:42.572390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:42.572356 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:44.198547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:44.198511 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:48.632103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:48.632067 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:50.555475 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:50.555418 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:50.714268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:50.714204 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:52.028785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.028745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:52.481842 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.481763 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm"] Nov 01 14:55:52.481997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.481976 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" containerID="cri-o://140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593" gracePeriod=30 Nov 01 14:55:52.497004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.496964 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:52.625588 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.625554 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:55:52.625767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.625621 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:55:52.757080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.757002 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8"] Nov 01 14:55:52.757606 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.757591 5473 cpu_manager.go:414] "containerMap: removing container" podUID="26179a37-0712-40ec-8d08-a824aa6fb6a2" containerName="revoke-keys" Nov 01 14:55:52.757654 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.757607 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26179a37-0712-40ec-8d08-a824aa6fb6a2" containerName="revoke-keys" Nov 01 14:55:52.757726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.757717 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="26179a37-0712-40ec-8d08-a824aa6fb6a2" containerName="revoke-keys" Nov 01 14:55:52.762776 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.762755 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.813356 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.813323 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8"] Nov 01 14:55:52.820486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.820454 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:55:52.820683 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.820606 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:55:52.839689 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.839644 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.839894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.839838 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.839983 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.839947 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72c7\" (UniqueName: \"kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.941797 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.941755 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.942024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.941893 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.942024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.941979 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72c7\" (UniqueName: \"kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.942653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.942632 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.944317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.944300 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:52.998780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:52.998751 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72c7\" (UniqueName: \"kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7\") pod \"maas-api-revoke-sub-1fc1f995-cbdq8\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:53.076159 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:53.076124 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:53.136101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:53.136072 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:53.298227 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:53.298186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:53.450664 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:55:53.450635 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3ddcea_4a38_42e3_9e73_34766b708211.slice/crio-a43face37177632ae8f71fbf57c070e9f8b3c55e87bb4222b97e735cb4146b95 WatchSource:0}: Error finding container a43face37177632ae8f71fbf57c070e9f8b3c55e87bb4222b97e735cb4146b95: Status 404 returned error can't find the container with id a43face37177632ae8f71fbf57c070e9f8b3c55e87bb4222b97e735cb4146b95 Nov 01 14:55:53.455768 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:53.455744 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8"] Nov 01 14:55:54.422820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:54.422789 5473 generic.go:356] "Generic (PLEG): container finished" podID="8c3ddcea-4a38-42e3-9e73-34766b708211" containerID="63d96ad6876e088e6a6ac530ac160fff3bfef87e744357cb80a496512085ce32" exitCode=0 Nov 01 14:55:54.423232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:54.422875 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" event={"ID":"8c3ddcea-4a38-42e3-9e73-34766b708211","Type":"ContainerDied","Data":"63d96ad6876e088e6a6ac530ac160fff3bfef87e744357cb80a496512085ce32"} Nov 01 14:55:54.423232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:54.422965 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" event={"ID":"8c3ddcea-4a38-42e3-9e73-34766b708211","Type":"ContainerStarted","Data":"a43face37177632ae8f71fbf57c070e9f8b3c55e87bb4222b97e735cb4146b95"} Nov 01 14:55:55.112655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.112613 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:55.571778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.571754 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:55.674029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.673996 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca\") pod \"8c3ddcea-4a38-42e3-9e73-34766b708211\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " Nov 01 14:55:55.674029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.674038 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca\") pod \"8c3ddcea-4a38-42e3-9e73-34766b708211\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " Nov 01 14:55:55.674282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.674131 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7\" (UniqueName: \"kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7\") pod \"8c3ddcea-4a38-42e3-9e73-34766b708211\" (UID: \"8c3ddcea-4a38-42e3-9e73-34766b708211\") " Nov 01 14:55:55.674478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.674452 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca" pod "8c3ddcea-4a38-42e3-9e73-34766b708211" (UID: "8c3ddcea-4a38-42e3-9e73-34766b708211"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:55:55.674759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.674738 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:55.676262 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.676232 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca" pod "8c3ddcea-4a38-42e3-9e73-34766b708211" (UID: "8c3ddcea-4a38-42e3-9e73-34766b708211"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:55:55.676369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.676272 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7" pod "8c3ddcea-4a38-42e3-9e73-34766b708211" (UID: "8c3ddcea-4a38-42e3-9e73-34766b708211"). InnerVolumeSpecName "kube-api-access-q72c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:55:55.775933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.775895 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3ddcea-4a38-42e3-9e73-34766b708211-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:55.775933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:55.775928 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72c7\" (UniqueName: \"kubernetes.io/projected/8c3ddcea-4a38-42e3-9e73-34766b708211-kube-api-access-q72c7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:56.438234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:56.438191 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" Nov 01 14:55:56.438234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:56.438202 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8" event={"ID":"8c3ddcea-4a38-42e3-9e73-34766b708211","Type":"ContainerDied","Data":"a43face37177632ae8f71fbf57c070e9f8b3c55e87bb4222b97e735cb4146b95"} Nov 01 14:55:56.438454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:56.438262 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a43face37177632ae8f71fbf57c070e9f8b3c55e87bb4222b97e735cb4146b95" Nov 01 14:55:57.753585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.753516 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:55:57.800769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.800739 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:57.903828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.903797 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.903892 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.903917 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.903939 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904013 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.903987 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904080 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904145 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904241 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904262 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:55:57.904378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904298 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert\") pod \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\" (UID: \"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce\") " Nov 01 14:55:57.904513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904424 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:55:57.904513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:55:57.904671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904650 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:55:57.904815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904767 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:55:57.905015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.904992 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:57.905116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.905017 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:57.905116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.905033 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:57.905116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.905047 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:57.905116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.905063 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:57.906151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.906130 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:55:57.906404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.906380 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:55:57.906507 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.906489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:55:57.906557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:57.906515 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz" pod "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" (UID: "d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce"). InnerVolumeSpecName "kube-api-access-lsrlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:55:58.005875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.005791 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:58.005875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.005823 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsrlz\" (UniqueName: \"kubernetes.io/projected/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-kube-api-access-lsrlz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:58.005875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.005834 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:58.005875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.005843 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:55:58.453892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.453851 5473 generic.go:356] "Generic (PLEG): container finished" podID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerID="140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593" exitCode=0 Nov 01 14:55:58.454068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.453907 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" Nov 01 14:55:58.454068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.453933 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" event={"ID":"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce","Type":"ContainerDied","Data":"140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593"} Nov 01 14:55:58.454068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.453991 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm" event={"ID":"d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce","Type":"ContainerDied","Data":"1fc2e5feb0bc7f6b07dea85aa5976ee96c9689d4f2885967cfae4c5c0f90f5b7"} Nov 01 14:55:58.454068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.454013 5473 scope.go:122] "RemoveContainer" containerID="140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593" Nov 01 14:55:58.464816 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.464796 5473 scope.go:122] "RemoveContainer" containerID="a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2" Nov 01 14:55:58.474137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.474119 5473 scope.go:122] "RemoveContainer" containerID="140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593" Nov 01 14:55:58.474439 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:55:58.474416 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593\": container with ID starting with 140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593 not found: ID does not exist" containerID="140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593" Nov 01 14:55:58.474502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.474450 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593"} err="failed to get container status \"140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593\": rpc error: code = NotFound desc = could not find container \"140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593\": container with ID starting with 140d39321325805bf3279fb148c5bcde7044e4f09c630a8bfe1292503b6d5593 not found: ID does not exist" Nov 01 14:55:58.474502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.474476 5473 scope.go:122] "RemoveContainer" containerID="a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2" Nov 01 14:55:58.474767 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:55:58.474746 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2\": container with ID starting with a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2 not found: ID does not exist" containerID="a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2" Nov 01 14:55:58.474826 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.474772 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2"} err="failed to get container status \"a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2\": rpc error: code = NotFound desc = could not find container \"a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2\": container with ID starting with a17dc110bcc7af5a75ee7ab35a20fa4e8a0c7c766f86bd8dc103329650c597b2 not found: ID does not exist" Nov 01 14:55:58.627080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.627042 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm"] Nov 01 14:55:58.676695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:58.676664 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0bb4b4-openshift-default-84bcd45c77-m2zjm"] Nov 01 14:55:59.085336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:59.085304 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" path="/var/lib/kubelet/pods/d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce/volumes" Nov 01 14:55:59.408640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:59.408565 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:55:59.906693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:55:59.906665 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm"] Nov 01 14:56:00.012599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:00.012566 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7"] Nov 01 14:56:00.271926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:00.271897 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:00.429693 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:00.429645 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:01.107273 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:56:01.107239 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:56:02.410671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:02.410615 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:02.574486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:02.574001 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:02.747840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:02.747719 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:04.907946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:04.907907 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" podUID="65f54644-cc0e-40fc-98ca-4a122472ed7f" containerName="payload-processing" containerID="cri-o://95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b" gracePeriod=25 Nov 01 14:56:05.013280 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.013235 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" podUID="80f51451-c201-4896-90ef-5e34d74e59ae" containerName="payload-pre-processing" containerID="cri-o://7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c" gracePeriod=25 Nov 01 14:56:05.181750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.181720 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:56:05.187028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.187002 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t\" (UniqueName: \"kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t\") pod \"65f54644-cc0e-40fc-98ca-4a122472ed7f\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " Nov 01 14:56:05.187172 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.187091 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume\") pod \"65f54644-cc0e-40fc-98ca-4a122472ed7f\" (UID: \"65f54644-cc0e-40fc-98ca-4a122472ed7f\") " Nov 01 14:56:05.187471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.187447 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume" pod "65f54644-cc0e-40fc-98ca-4a122472ed7f" (UID: "65f54644-cc0e-40fc-98ca-4a122472ed7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:56:05.189699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.189672 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t" pod "65f54644-cc0e-40fc-98ca-4a122472ed7f" (UID: "65f54644-cc0e-40fc-98ca-4a122472ed7f"). InnerVolumeSpecName "kube-api-access-sw96t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:05.288429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.288391 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw96t\" (UniqueName: \"kubernetes.io/projected/65f54644-cc0e-40fc-98ca-4a122472ed7f-kube-api-access-sw96t\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:05.288429 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.288430 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f54644-cc0e-40fc-98ca-4a122472ed7f-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:05.305107 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.305079 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:56:05.376428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.376391 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:05.388901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.388880 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume\") pod \"80f51451-c201-4896-90ef-5e34d74e59ae\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " Nov 01 14:56:05.389020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.388983 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr\" (UniqueName: \"kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr\") pod \"80f51451-c201-4896-90ef-5e34d74e59ae\" (UID: \"80f51451-c201-4896-90ef-5e34d74e59ae\") " Nov 01 14:56:05.389303 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.389281 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume" pod "80f51451-c201-4896-90ef-5e34d74e59ae" (UID: "80f51451-c201-4896-90ef-5e34d74e59ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:56:05.391055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.391025 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr" pod "80f51451-c201-4896-90ef-5e34d74e59ae" (UID: "80f51451-c201-4896-90ef-5e34d74e59ae"). InnerVolumeSpecName "kube-api-access-2rbkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:05.489972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.489894 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f51451-c201-4896-90ef-5e34d74e59ae-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:05.489972 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.489921 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rbkr\" (UniqueName: \"kubernetes.io/projected/80f51451-c201-4896-90ef-5e34d74e59ae-kube-api-access-2rbkr\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:05.531756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.531725 5473 generic.go:356] "Generic (PLEG): container finished" podID="65f54644-cc0e-40fc-98ca-4a122472ed7f" containerID="95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b" exitCode=0 Nov 01 14:56:05.531934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.531767 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" Nov 01 14:56:05.531934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.531807 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" event={"ID":"65f54644-cc0e-40fc-98ca-4a122472ed7f","Type":"ContainerDied","Data":"95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b"} Nov 01 14:56:05.531934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.531856 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm" event={"ID":"65f54644-cc0e-40fc-98ca-4a122472ed7f","Type":"ContainerDied","Data":"c9d4f77d1586e687443ffe8e43791f4c27f9a1e5d86e675a250c90776065b985"} Nov 01 14:56:05.531934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.531888 5473 scope.go:122] "RemoveContainer" containerID="95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b" Nov 01 14:56:05.533511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.533490 5473 generic.go:356] "Generic (PLEG): container finished" podID="80f51451-c201-4896-90ef-5e34d74e59ae" containerID="7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c" exitCode=0 Nov 01 14:56:05.533615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.533545 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" Nov 01 14:56:05.533615 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.533579 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" event={"ID":"80f51451-c201-4896-90ef-5e34d74e59ae","Type":"ContainerDied","Data":"7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c"} Nov 01 14:56:05.533707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.533631 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7" event={"ID":"80f51451-c201-4896-90ef-5e34d74e59ae","Type":"ContainerDied","Data":"48013fb83f8ec91d303ad76c00d1fc9daab8d54c281740741fe89535cd9463b9"} Nov 01 14:56:05.542492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.542429 5473 scope.go:122] "RemoveContainer" containerID="95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b" Nov 01 14:56:05.542788 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:56:05.542767 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b\": container with ID starting with 95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b not found: ID does not exist" containerID="95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b" Nov 01 14:56:05.542885 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.542794 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b"} err="failed to get container status \"95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b\": rpc error: code = NotFound desc = could not find container \"95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b\": container with ID starting with 95b7599ca7a5c1f612f8284d8647880477053ab7df29ca8416b6022a79d0346b not found: ID does not exist" Nov 01 14:56:05.542885 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.542818 5473 scope.go:122] "RemoveContainer" containerID="7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c" Nov 01 14:56:05.551920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.551903 5473 scope.go:122] "RemoveContainer" containerID="7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c" Nov 01 14:56:05.552198 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:56:05.552178 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c\": container with ID starting with 7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c not found: ID does not exist" containerID="7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c" Nov 01 14:56:05.552295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.552205 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c"} err="failed to get container status \"7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c\": rpc error: code = NotFound desc = could not find container \"7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c\": container with ID starting with 7fda436f0a80aef780620a43dc5bb800fd0e70d8569cc358f7bb7b412e59272c not found: ID does not exist" Nov 01 14:56:05.697937 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.697902 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm"] Nov 01 14:56:05.700992 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.700971 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-305290-6d9d645795-xmfgm"] Nov 01 14:56:05.809766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.809735 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7"] Nov 01 14:56:05.818428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.818400 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-305290-845cc4cd9f-wttm7"] Nov 01 14:56:05.821335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821315 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22"] Nov 01 14:56:05.821834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821821 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" Nov 01 14:56:05.821880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821835 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" Nov 01 14:56:05.821880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821849 5473 cpu_manager.go:414] "containerMap: removing container" podUID="65f54644-cc0e-40fc-98ca-4a122472ed7f" containerName="payload-processing" Nov 01 14:56:05.821880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821854 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f54644-cc0e-40fc-98ca-4a122472ed7f" containerName="payload-processing" Nov 01 14:56:05.821880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821876 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" Nov 01 14:56:05.821880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821881 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821888 5473 cpu_manager.go:414] "containerMap: removing container" podUID="80f51451-c201-4896-90ef-5e34d74e59ae" containerName="payload-pre-processing" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821893 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f51451-c201-4896-90ef-5e34d74e59ae" containerName="payload-pre-processing" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821916 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8c3ddcea-4a38-42e3-9e73-34766b708211" containerName="revoke-keys" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821921 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3ddcea-4a38-42e3-9e73-34766b708211" containerName="revoke-keys" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.821999 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.822005 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f51451-c201-4896-90ef-5e34d74e59ae" containerName="payload-pre-processing" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.822016 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3ddcea-4a38-42e3-9e73-34766b708211" containerName="revoke-keys" Nov 01 14:56:05.822046 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.822025 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f54644-cc0e-40fc-98ca-4a122472ed7f" containerName="payload-processing" Nov 01 14:56:05.826765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.826749 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.893492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893462 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.893780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893507 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.893780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893525 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.893780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893656 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.893780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893685 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.893780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893762 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc886\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.894069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893819 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.894069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893853 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.894069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.893936 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.914952 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.914922 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22"] Nov 01 14:56:05.994910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.994874 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.994910 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.994910 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995149 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.994970 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995149 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.994999 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995149 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995062 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc886\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995150 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995277 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995342 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995425 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995452 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995661 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995639 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995735 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995711 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995831 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.995901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.995877 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.997502 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.997482 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:05.997595 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:05.997548 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:06.052988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.052951 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:06.053152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.053066 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc886\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:06.140192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.140108 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:06.560467 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:56:06.560439 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ab68bc_e2ea_4021_974b_86ec3260e5a7.slice/crio-6838e8382c39b2c942d75055f59a69378a67f1fdee4a08a6ccf8b3f40088dedd WatchSource:0}: Error finding container 6838e8382c39b2c942d75055f59a69378a67f1fdee4a08a6ccf8b3f40088dedd: Status 404 returned error can't find the container with id 6838e8382c39b2c942d75055f59a69378a67f1fdee4a08a6ccf8b3f40088dedd Nov 01 14:56:06.562829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.562794 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:56:06.562938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.562878 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:56:06.562938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.562922 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:56:06.563982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:06.563954 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22"] Nov 01 14:56:07.087055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:07.087016 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f54644-cc0e-40fc-98ca-4a122472ed7f" path="/var/lib/kubelet/pods/65f54644-cc0e-40fc-98ca-4a122472ed7f/volumes" Nov 01 14:56:07.088283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:07.088257 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f51451-c201-4896-90ef-5e34d74e59ae" path="/var/lib/kubelet/pods/80f51451-c201-4896-90ef-5e34d74e59ae/volumes" Nov 01 14:56:07.293700 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:07.293666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:07.555272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:07.555231 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" event={"ID":"90ab68bc-e2ea-4021-974b-86ec3260e5a7","Type":"ContainerStarted","Data":"8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b"} Nov 01 14:56:07.555460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:07.555287 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" event={"ID":"90ab68bc-e2ea-4021-974b-86ec3260e5a7","Type":"ContainerStarted","Data":"6838e8382c39b2c942d75055f59a69378a67f1fdee4a08a6ccf8b3f40088dedd"} Nov 01 14:56:07.777190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:07.777132 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podStartSLOduration=2.777118269 podStartE2EDuration="2.777118269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:07.774345275 +0000 UTC m=+2306.832497125" watchObservedRunningTime="2026-11-01 14:56:07.777118269 +0000 UTC m=+2306.835270118" Nov 01 14:56:08.149239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:08.148249 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:08.149239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:08.148385 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 14:56:08.149239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:08.148433 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 14:56:09.143640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:09.143600 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 14:56:09.143927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:09.143897 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 14:56:10.143703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:10.143667 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 14:56:10.144372 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:10.144335 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 14:56:11.149361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:11.149326 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 14:56:11.149866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:11.149379 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 14:56:11.933240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:11.930483 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:12.142641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:12.142603 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 14:56:12.142943 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:12.142913 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 14:56:13.141321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.141283 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 01 14:56:13.141777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.141362 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 01 14:56:13.588565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.588532 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m"] Nov 01 14:56:13.589430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.589407 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c12e9c-f109-4dbb-8ba2-70a7b978b7ce" containerName="istio-proxy" Nov 01 14:56:13.595496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.595469 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.664962 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.664930 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m"] Nov 01 14:56:13.705332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.705295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.705516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.705362 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.705516 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.705398 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d88f\" (UniqueName: \"kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.708874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.708829 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:56:13.709027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.709010 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:56:13.806844 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.806810 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.807056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.806898 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.807056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.806955 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d88f\" (UniqueName: \"kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.807865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.807833 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.809357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.809334 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.863997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.863928 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d88f\" (UniqueName: \"kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f\") pod \"maas-api-revoke-sub-16523292-bl44m\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:13.914094 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:13.914057 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:14.145133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.145060 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:14.145537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.145519 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:14.146276 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.146253 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 14:56:14.285240 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:56:14.285190 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e18c4f_cb89_4944_979a_d06891fadce9.slice/crio-4f416eaa578c087345696da3db2d16cce2e462f9db7a91630b2e96f2206505d0 WatchSource:0}: Error finding container 4f416eaa578c087345696da3db2d16cce2e462f9db7a91630b2e96f2206505d0: Status 404 returned error can't find the container with id 4f416eaa578c087345696da3db2d16cce2e462f9db7a91630b2e96f2206505d0 Nov 01 14:56:14.314812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.314787 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m"] Nov 01 14:56:14.607719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.607682 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" event={"ID":"38e18c4f-cb89-4944-979a-d06891fadce9","Type":"ContainerStarted","Data":"361d4a671b5be6320075edfa1e3e273f69eac521a20ff9925ebe0be41ccb6708"} Nov 01 14:56:14.607895 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.607749 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" event={"ID":"38e18c4f-cb89-4944-979a-d06891fadce9","Type":"ContainerStarted","Data":"4f416eaa578c087345696da3db2d16cce2e462f9db7a91630b2e96f2206505d0"} Nov 01 14:56:14.724287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:14.724169 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" podStartSLOduration=1.724155406 podStartE2EDuration="1.724155406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:14.720936433 +0000 UTC m=+2313.779088307" watchObservedRunningTime="2026-11-01 14:56:14.724155406 +0000 UTC m=+2313.782307256" Nov 01 14:56:15.616608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:15.616570 5473 generic.go:356] "Generic (PLEG): container finished" podID="38e18c4f-cb89-4944-979a-d06891fadce9" containerID="361d4a671b5be6320075edfa1e3e273f69eac521a20ff9925ebe0be41ccb6708" exitCode=0 Nov 01 14:56:15.616985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:15.616653 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" event={"ID":"38e18c4f-cb89-4944-979a-d06891fadce9","Type":"ContainerDied","Data":"361d4a671b5be6320075edfa1e3e273f69eac521a20ff9925ebe0be41ccb6708"} Nov 01 14:56:16.515328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.515289 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:16.765921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.765854 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:16.848527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.848494 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f\" (UniqueName: \"kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f\") pod \"38e18c4f-cb89-4944-979a-d06891fadce9\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " Nov 01 14:56:16.848707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.848549 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca\") pod \"38e18c4f-cb89-4944-979a-d06891fadce9\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " Nov 01 14:56:16.848760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.848733 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca\") pod \"38e18c4f-cb89-4944-979a-d06891fadce9\" (UID: \"38e18c4f-cb89-4944-979a-d06891fadce9\") " Nov 01 14:56:16.849072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.849044 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca" pod "38e18c4f-cb89-4944-979a-d06891fadce9" (UID: "38e18c4f-cb89-4944-979a-d06891fadce9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:56:16.850657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.850623 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f" pod "38e18c4f-cb89-4944-979a-d06891fadce9" (UID: "38e18c4f-cb89-4944-979a-d06891fadce9"). InnerVolumeSpecName "kube-api-access-5d88f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:16.850773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.850676 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca" pod "38e18c4f-cb89-4944-979a-d06891fadce9" (UID: "38e18c4f-cb89-4944-979a-d06891fadce9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:56:16.949787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.949754 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:16.949787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.949784 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d88f\" (UniqueName: \"kubernetes.io/projected/38e18c4f-cb89-4944-979a-d06891fadce9-kube-api-access-5d88f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:16.949787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:16.949795 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e18c4f-cb89-4944-979a-d06891fadce9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:17.632200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:17.632165 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" Nov 01 14:56:17.632200 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:17.632188 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m" event={"ID":"38e18c4f-cb89-4944-979a-d06891fadce9","Type":"ContainerDied","Data":"4f416eaa578c087345696da3db2d16cce2e462f9db7a91630b2e96f2206505d0"} Nov 01 14:56:17.632447 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:17.632260 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f416eaa578c087345696da3db2d16cce2e462f9db7a91630b2e96f2206505d0" Nov 01 14:56:22.498501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:22.498434 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:26.916051 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:26.916004 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:28.029515 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.029427 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw"] Nov 01 14:56:28.030272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.030247 5473 cpu_manager.go:414] "containerMap: removing container" podUID="38e18c4f-cb89-4944-979a-d06891fadce9" containerName="revoke-keys" Nov 01 14:56:28.030272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.030272 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e18c4f-cb89-4944-979a-d06891fadce9" containerName="revoke-keys" Nov 01 14:56:28.030451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.030438 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e18c4f-cb89-4944-979a-d06891fadce9" containerName="revoke-keys" Nov 01 14:56:28.033868 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.033846 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.085875 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.085836 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw"] Nov 01 14:56:28.131906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.131873 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:56:28.131906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.131899 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:56:28.135759 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.135711 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p475p\" (UniqueName: \"kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.135870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.135848 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.136028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.136008 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.237174 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.237135 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.237353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.237228 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p475p\" (UniqueName: \"kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.237353 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.237310 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.237942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.237914 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.239579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.239561 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.294949 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.294904 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p475p\" (UniqueName: \"kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p\") pod \"maas-api-revoke-sub-0c0b0251-fplrw\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.347428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.347389 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:28.718475 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:56:28.718440 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1f2f6d_22af_4c28_99d4_59c1ffb2d655.slice/crio-579b6065fa0174c96588be9fdfa191057e863eacbbf43d365df265306428e661 WatchSource:0}: Error finding container 579b6065fa0174c96588be9fdfa191057e863eacbbf43d365df265306428e661: Status 404 returned error can't find the container with id 579b6065fa0174c96588be9fdfa191057e863eacbbf43d365df265306428e661 Nov 01 14:56:28.721870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.721844 5473 generic.go:356] "Generic (PLEG): container finished" podID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerID="ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d" exitCode=0 Nov 01 14:56:28.721968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.721881 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" event={"ID":"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32","Type":"ContainerDied","Data":"ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d"} Nov 01 14:56:28.722244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.722186 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw"] Nov 01 14:56:28.722357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.722292 5473 scope.go:122] "RemoveContainer" containerID="ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d" Nov 01 14:56:28.724182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.724147 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:56:28.724273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.724259 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:56:28.724318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:28.724308 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:56:29.754894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:29.754850 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" event={"ID":"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32","Type":"ContainerStarted","Data":"657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b"} Nov 01 14:56:29.757237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:29.757192 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" event={"ID":"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655","Type":"ContainerStarted","Data":"86614e1c9fe82341f4ec9c06788b7ef34ae41f9726f44c91fd5ae98168364983"} Nov 01 14:56:29.757390 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:29.757260 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" event={"ID":"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655","Type":"ContainerStarted","Data":"579b6065fa0174c96588be9fdfa191057e863eacbbf43d365df265306428e661"} Nov 01 14:56:30.007241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:30.007121 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" podStartSLOduration=3.007107721 podStartE2EDuration="3.007107721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:30.005381129 +0000 UTC m=+2329.063532978" watchObservedRunningTime="2026-11-01 14:56:30.007107721 +0000 UTC m=+2329.065259573" Nov 01 14:56:30.091783 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:30.091746 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:56:30.093165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:30.093138 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 14:56:30.093314 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:30.093189 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 14:56:30.766920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:30.766886 5473 generic.go:356] "Generic (PLEG): container finished" podID="bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" containerID="86614e1c9fe82341f4ec9c06788b7ef34ae41f9726f44c91fd5ae98168364983" exitCode=0 Nov 01 14:56:30.767370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:30.766986 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" event={"ID":"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655","Type":"ContainerDied","Data":"86614e1c9fe82341f4ec9c06788b7ef34ae41f9726f44c91fd5ae98168364983"} Nov 01 14:56:31.091306 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:31.091210 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 14:56:31.091455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:31.091309 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 14:56:31.107685 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:56:31.107649 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:56:31.543241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:31.540083 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:31.957711 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:31.957686 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:32.088812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.088732 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca\") pod \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " Nov 01 14:56:32.088812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.088775 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p\" (UniqueName: \"kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p\") pod \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " Nov 01 14:56:32.089060 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.088837 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca\") pod \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\" (UID: \"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655\") " Nov 01 14:56:32.089381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.089340 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca" pod "bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" (UID: "bd1f2f6d-22af-4c28-99d4-59c1ffb2d655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:56:32.091014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.090988 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p" pod "bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" (UID: "bd1f2f6d-22af-4c28-99d4-59c1ffb2d655"). InnerVolumeSpecName "kube-api-access-p475p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:32.091181 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.091026 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 14:56:32.091334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.091308 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 14:56:32.092210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.092186 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca" pod "bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" (UID: "bd1f2f6d-22af-4c28-99d4-59c1ffb2d655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:56:32.190491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.190445 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:32.190491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.190487 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p475p\" (UniqueName: \"kubernetes.io/projected/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-kube-api-access-p475p\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:32.190491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.190502 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:32.783043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.783014 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" Nov 01 14:56:32.783232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.783049 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw" event={"ID":"bd1f2f6d-22af-4c28-99d4-59c1ffb2d655","Type":"ContainerDied","Data":"579b6065fa0174c96588be9fdfa191057e863eacbbf43d365df265306428e661"} Nov 01 14:56:32.783232 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:32.783127 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="579b6065fa0174c96588be9fdfa191057e863eacbbf43d365df265306428e661" Nov 01 14:56:33.018304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:33.018273 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:56:33.091567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:33.091483 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 14:56:33.091567 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:33.091554 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 14:56:34.091210 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:34.091172 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Nov 01 14:56:34.091653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:34.091262 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Nov 01 14:56:35.096088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:35.096059 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:56:35.097198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:35.097174 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:56:36.038099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.038069 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts"] Nov 01 14:56:36.038296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.038275 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" podUID="56a13785-e4fa-4d14-b930-7a447344288a" containerName="discovery" containerID="cri-o://10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9" gracePeriod=30 Nov 01 14:56:36.333626 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.333596 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:56:36.439737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.439710 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.439880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.439747 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.439880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.439773 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.439880 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.439843 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.440029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.439878 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.440029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.439936 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.440133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.440086 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert\") pod \"56a13785-e4fa-4d14-b930-7a447344288a\" (UID: \"56a13785-e4fa-4d14-b930-7a447344288a\") " Nov 01 14:56:36.440471 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.440311 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:56:36.440741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.440701 5473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-ca-configmap\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.442336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.442295 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:56:36.442546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.442526 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "kube-api-access-l9s7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:36.442637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.442568 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:56:36.442637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.442577 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:56:36.442637 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.442570 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:36.442760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.442699 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert" pod "56a13785-e4fa-4d14-b930-7a447344288a" (UID: "56a13785-e4fa-4d14-b930-7a447344288a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:56:36.542027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.541993 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.542027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.542022 5473 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-kubeconfig\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.542027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.542033 5473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-istio-csr-dns-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.542329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.542043 5473 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/56a13785-e4fa-4d14-b930-7a447344288a-local-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.542329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.542052 5473 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/56a13785-e4fa-4d14-b930-7a447344288a-cacerts\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.542329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.542063 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9s7s\" (UniqueName: \"kubernetes.io/projected/56a13785-e4fa-4d14-b930-7a447344288a-kube-api-access-l9s7s\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:36.822773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.822737 5473 generic.go:356] "Generic (PLEG): container finished" podID="56a13785-e4fa-4d14-b930-7a447344288a" containerID="10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9" exitCode=0 Nov 01 14:56:36.822944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.822789 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" Nov 01 14:56:36.822944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.822825 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" event={"ID":"56a13785-e4fa-4d14-b930-7a447344288a","Type":"ContainerDied","Data":"10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9"} Nov 01 14:56:36.822944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.822921 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts" event={"ID":"56a13785-e4fa-4d14-b930-7a447344288a","Type":"ContainerDied","Data":"92f0d9a996b310bc8fa13eb754c51479d70910f5c34616bd65049c158aa54fd5"} Nov 01 14:56:36.823106 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.822950 5473 scope.go:122] "RemoveContainer" containerID="10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9" Nov 01 14:56:36.834760 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.834739 5473 scope.go:122] "RemoveContainer" containerID="10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9" Nov 01 14:56:36.835024 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:56:36.835007 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9\": container with ID starting with 10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9 not found: ID does not exist" containerID="10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9" Nov 01 14:56:36.835083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.835031 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9"} err="failed to get container status \"10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9\": rpc error: code = NotFound desc = could not find container \"10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9\": container with ID starting with 10c136dc331843fddc936eb785e18172290b4ad2ca414d8158068144b7705be9 not found: ID does not exist" Nov 01 14:56:36.988269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:36.988242 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts"] Nov 01 14:56:37.038171 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:37.038144 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lzcts"] Nov 01 14:56:37.082069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:37.081998 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a13785-e4fa-4d14-b930-7a447344288a" path="/var/lib/kubelet/pods/56a13785-e4fa-4d14-b930-7a447344288a/volumes" Nov 01 14:56:37.585063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:37.585032 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:41.897856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:41.897822 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:46.336179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:46.336145 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:48.693856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.693822 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd"] Nov 01 14:56:48.694412 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.694397 5473 cpu_manager.go:414] "containerMap: removing container" podUID="56a13785-e4fa-4d14-b930-7a447344288a" containerName="discovery" Nov 01 14:56:48.694473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.694413 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a13785-e4fa-4d14-b930-7a447344288a" containerName="discovery" Nov 01 14:56:48.694473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.694431 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" containerName="revoke-keys" Nov 01 14:56:48.694473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.694437 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" containerName="revoke-keys" Nov 01 14:56:48.694581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.694527 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a13785-e4fa-4d14-b930-7a447344288a" containerName="discovery" Nov 01 14:56:48.694581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.694537 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" containerName="revoke-keys" Nov 01 14:56:48.699753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.699733 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.751500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.751470 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd"] Nov 01 14:56:48.780553 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.780524 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwjq\" (UniqueName: \"kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.780678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.780637 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.780727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.780687 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.796926 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.796906 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:56:48.797066 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.797050 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:56:48.881899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.881868 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.882065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.881986 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwjq\" (UniqueName: \"kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.882065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.882043 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.882656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.882634 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.884332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.884312 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:48.938963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:48.938937 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwjq\" (UniqueName: \"kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq\") pod \"maas-api-revoke-sub-0a5af36c-mhgpd\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:49.014065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:49.014042 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:49.381857 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:56:49.381814 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60feb8c_9858_4c60_9001_bd7dd503c589.slice/crio-b8de50357a41ff12cc5649fef6a291b4bc17914d4e15dac31b68419622c7d84c WatchSource:0}: Error finding container b8de50357a41ff12cc5649fef6a291b4bc17914d4e15dac31b68419622c7d84c: Status 404 returned error can't find the container with id b8de50357a41ff12cc5649fef6a291b4bc17914d4e15dac31b68419622c7d84c Nov 01 14:56:49.385778 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:49.385756 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd"] Nov 01 14:56:49.913860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:49.913782 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" event={"ID":"f60feb8c-9858-4c60-9001-bd7dd503c589","Type":"ContainerStarted","Data":"c620eceaaccf492fe952b1d3d57eea8d908a1a0a754deaf6b60bdfe7221be22b"} Nov 01 14:56:49.913860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:49.913842 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" event={"ID":"f60feb8c-9858-4c60-9001-bd7dd503c589","Type":"ContainerStarted","Data":"b8de50357a41ff12cc5649fef6a291b4bc17914d4e15dac31b68419622c7d84c"} Nov 01 14:56:50.031593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:50.031542 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" podStartSLOduration=2.031527848 podStartE2EDuration="2.031527848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:56:50.029130074 +0000 UTC m=+2349.087281926" watchObservedRunningTime="2026-11-01 14:56:50.031527848 +0000 UTC m=+2349.089679697" Nov 01 14:56:50.146298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:50.146270 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:50.930731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:50.930682 5473 generic.go:356] "Generic (PLEG): container finished" podID="f60feb8c-9858-4c60-9001-bd7dd503c589" containerID="c620eceaaccf492fe952b1d3d57eea8d908a1a0a754deaf6b60bdfe7221be22b" exitCode=0 Nov 01 14:56:50.931250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:50.931070 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" event={"ID":"f60feb8c-9858-4c60-9001-bd7dd503c589","Type":"ContainerDied","Data":"c620eceaaccf492fe952b1d3d57eea8d908a1a0a754deaf6b60bdfe7221be22b"} Nov 01 14:56:52.086369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.086344 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:52.218912 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.218841 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca\") pod \"f60feb8c-9858-4c60-9001-bd7dd503c589\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " Nov 01 14:56:52.219083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.218996 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq\" (UniqueName: \"kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq\") pod \"f60feb8c-9858-4c60-9001-bd7dd503c589\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " Nov 01 14:56:52.219083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.219043 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca\") pod \"f60feb8c-9858-4c60-9001-bd7dd503c589\" (UID: \"f60feb8c-9858-4c60-9001-bd7dd503c589\") " Nov 01 14:56:52.219287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.219260 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca" pod "f60feb8c-9858-4c60-9001-bd7dd503c589" (UID: "f60feb8c-9858-4c60-9001-bd7dd503c589"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:56:52.219557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.219535 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:52.221269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.221205 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca" pod "f60feb8c-9858-4c60-9001-bd7dd503c589" (UID: "f60feb8c-9858-4c60-9001-bd7dd503c589"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:56:52.221456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.221439 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq" pod "f60feb8c-9858-4c60-9001-bd7dd503c589" (UID: "f60feb8c-9858-4c60-9001-bd7dd503c589"). InnerVolumeSpecName "kube-api-access-nwwjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:56:52.320775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.320749 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwjq\" (UniqueName: \"kubernetes.io/projected/f60feb8c-9858-4c60-9001-bd7dd503c589-kube-api-access-nwwjq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:52.320775 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.320775 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f60feb8c-9858-4c60-9001-bd7dd503c589-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:56:52.369206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.369180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:52.947420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.947394 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" Nov 01 14:56:52.947627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.947422 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd" event={"ID":"f60feb8c-9858-4c60-9001-bd7dd503c589","Type":"ContainerDied","Data":"b8de50357a41ff12cc5649fef6a291b4bc17914d4e15dac31b68419622c7d84c"} Nov 01 14:56:52.947627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:52.947495 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8de50357a41ff12cc5649fef6a291b4bc17914d4e15dac31b68419622c7d84c" Nov 01 14:56:55.489920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:55.489887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:55.661995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:55.661963 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:56.317618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:56.317589 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:57.103818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:57.103766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:57.670993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:57.670955 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:56:57.836752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:56:57.836709 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:01.022479 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:01.022422 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:01.108368 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:57:01.108337 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:57:02.935190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:02.935112 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:05.604588 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:05.604557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:07.249154 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.249120 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:07.527420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.527384 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht"] Nov 01 14:57:07.527948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.527935 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f60feb8c-9858-4c60-9001-bd7dd503c589" containerName="revoke-keys" Nov 01 14:57:07.528014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.527949 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60feb8c-9858-4c60-9001-bd7dd503c589" containerName="revoke-keys" Nov 01 14:57:07.528076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.528066 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60feb8c-9858-4c60-9001-bd7dd503c589" containerName="revoke-keys" Nov 01 14:57:07.531530 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.531514 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.583976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.583944 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht"] Nov 01 14:57:07.603025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.602994 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gd2n\" (UniqueName: \"kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.603127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.603055 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.603206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.603186 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.629870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.629842 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:57:07.629973 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.629927 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:57:07.704510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.704487 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.704627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.704590 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gd2n\" (UniqueName: \"kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.704627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.704616 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.705242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.705192 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.706818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.706790 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.761743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.761714 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gd2n\" (UniqueName: \"kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n\") pod \"maas-api-revoke-sub-cdc6585d-b9hht\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:07.845456 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:07.845402 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:08.213795 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:08.213763 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015329eb_fabc_4993_a9f1_62f31728cc4f.slice/crio-a4c202b5bd3b150a003bfaebab90a09591bedab59a5e11f445a4a0caf5fdb8b9 WatchSource:0}: Error finding container a4c202b5bd3b150a003bfaebab90a09591bedab59a5e11f445a4a0caf5fdb8b9: Status 404 returned error can't find the container with id a4c202b5bd3b150a003bfaebab90a09591bedab59a5e11f445a4a0caf5fdb8b9 Nov 01 14:57:08.217562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:08.217537 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht"] Nov 01 14:57:09.065478 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:09.065445 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" event={"ID":"015329eb-fabc-4993-a9f1-62f31728cc4f","Type":"ContainerStarted","Data":"709745cb877db132a2f17bb0b6a88fd67e39a25a3d6096ce9605f4b6397209b2"} Nov 01 14:57:09.065863 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:09.065498 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" event={"ID":"015329eb-fabc-4993-a9f1-62f31728cc4f","Type":"ContainerStarted","Data":"a4c202b5bd3b150a003bfaebab90a09591bedab59a5e11f445a4a0caf5fdb8b9"} Nov 01 14:57:09.206721 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:09.206673 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" podStartSLOduration=2.206659924 podStartE2EDuration="2.206659924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:09.179483724 +0000 UTC m=+2368.237635573" watchObservedRunningTime="2026-11-01 14:57:09.206659924 +0000 UTC m=+2368.264811774" Nov 01 14:57:09.438183 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:09.438103 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:10.074284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.074248 5473 generic.go:356] "Generic (PLEG): container finished" podID="015329eb-fabc-4993-a9f1-62f31728cc4f" containerID="709745cb877db132a2f17bb0b6a88fd67e39a25a3d6096ce9605f4b6397209b2" exitCode=0 Nov 01 14:57:10.074667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.074305 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" event={"ID":"015329eb-fabc-4993-a9f1-62f31728cc4f","Type":"ContainerDied","Data":"709745cb877db132a2f17bb0b6a88fd67e39a25a3d6096ce9605f4b6397209b2"} Nov 01 14:57:10.188501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.188472 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk"] Nov 01 14:57:10.192591 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.192573 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.231256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.231232 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.231386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.231281 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.231386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.231359 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfl42\" (UniqueName: \"kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.245032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.245003 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk"] Nov 01 14:57:10.332448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.332374 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.332448 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.332428 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.332638 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.332459 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfl42\" (UniqueName: \"kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.333058 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.333031 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.334653 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.334628 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.389413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.389384 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfl42\" (UniqueName: \"kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42\") pod \"maas-api-revoke-sub-363ecad6-7ntjk\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.491547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.491517 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:10.505865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.505840 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:10.886034 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:10.886004 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e89169b_766a_413b_a844_bafa803a4911.slice/crio-56411f6f8f03fd7a004a4b596873815f7a9b64c8aa53c74115b9d76c5852d6be WatchSource:0}: Error finding container 56411f6f8f03fd7a004a4b596873815f7a9b64c8aa53c74115b9d76c5852d6be: Status 404 returned error can't find the container with id 56411f6f8f03fd7a004a4b596873815f7a9b64c8aa53c74115b9d76c5852d6be Nov 01 14:57:10.890257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:10.890233 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk"] Nov 01 14:57:11.087301 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.086262 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" event={"ID":"9e89169b-766a-413b-a844-bafa803a4911","Type":"ContainerStarted","Data":"56411f6f8f03fd7a004a4b596873815f7a9b64c8aa53c74115b9d76c5852d6be"} Nov 01 14:57:11.225352 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.225326 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:11.346452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.346422 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca\") pod \"015329eb-fabc-4993-a9f1-62f31728cc4f\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " Nov 01 14:57:11.346604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.346481 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca\") pod \"015329eb-fabc-4993-a9f1-62f31728cc4f\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " Nov 01 14:57:11.346604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.346599 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n\" (UniqueName: \"kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n\") pod \"015329eb-fabc-4993-a9f1-62f31728cc4f\" (UID: \"015329eb-fabc-4993-a9f1-62f31728cc4f\") " Nov 01 14:57:11.346853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.346825 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca" pod "015329eb-fabc-4993-a9f1-62f31728cc4f" (UID: "015329eb-fabc-4993-a9f1-62f31728cc4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:57:11.347414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.347062 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:11.348995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.348958 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n" pod "015329eb-fabc-4993-a9f1-62f31728cc4f" (UID: "015329eb-fabc-4993-a9f1-62f31728cc4f"). InnerVolumeSpecName "kube-api-access-4gd2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:57:11.349792 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.349767 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca" pod "015329eb-fabc-4993-a9f1-62f31728cc4f" (UID: "015329eb-fabc-4993-a9f1-62f31728cc4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:57:11.447840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.447763 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015329eb-fabc-4993-a9f1-62f31728cc4f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:11.447840 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:11.447791 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gd2n\" (UniqueName: \"kubernetes.io/projected/015329eb-fabc-4993-a9f1-62f31728cc4f-kube-api-access-4gd2n\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:12.095480 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:12.095450 5473 generic.go:356] "Generic (PLEG): container finished" podID="9e89169b-766a-413b-a844-bafa803a4911" containerID="6dce77dc5fb6363067bb935bafaca06358066e42dc4a413f4a2997f23be7cb36" exitCode=0 Nov 01 14:57:12.095873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:12.095529 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" event={"ID":"9e89169b-766a-413b-a844-bafa803a4911","Type":"ContainerDied","Data":"6dce77dc5fb6363067bb935bafaca06358066e42dc4a413f4a2997f23be7cb36"} Nov 01 14:57:12.096889 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:12.096861 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" Nov 01 14:57:12.097006 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:12.096907 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht" event={"ID":"015329eb-fabc-4993-a9f1-62f31728cc4f","Type":"ContainerDied","Data":"a4c202b5bd3b150a003bfaebab90a09591bedab59a5e11f445a4a0caf5fdb8b9"} Nov 01 14:57:12.097006 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:12.096949 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4c202b5bd3b150a003bfaebab90a09591bedab59a5e11f445a4a0caf5fdb8b9" Nov 01 14:57:13.254090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.254066 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:13.367072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.366978 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42\" (UniqueName: \"kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42\") pod \"9e89169b-766a-413b-a844-bafa803a4911\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " Nov 01 14:57:13.367252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.367131 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca\") pod \"9e89169b-766a-413b-a844-bafa803a4911\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " Nov 01 14:57:13.367252 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.367245 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca\") pod \"9e89169b-766a-413b-a844-bafa803a4911\" (UID: \"9e89169b-766a-413b-a844-bafa803a4911\") " Nov 01 14:57:13.367782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.367734 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca" pod "9e89169b-766a-413b-a844-bafa803a4911" (UID: "9e89169b-766a-413b-a844-bafa803a4911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:57:13.369254 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.369226 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca" pod "9e89169b-766a-413b-a844-bafa803a4911" (UID: "9e89169b-766a-413b-a844-bafa803a4911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:57:13.369519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.369489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42" pod "9e89169b-766a-413b-a844-bafa803a4911" (UID: "9e89169b-766a-413b-a844-bafa803a4911"). InnerVolumeSpecName "kube-api-access-bfl42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:57:13.468617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.468587 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfl42\" (UniqueName: \"kubernetes.io/projected/9e89169b-766a-413b-a844-bafa803a4911-kube-api-access-bfl42\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:13.468617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.468617 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e89169b-766a-413b-a844-bafa803a4911-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:13.468758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:13.468632 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e89169b-766a-413b-a844-bafa803a4911-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:14.111372 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:14.111343 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" Nov 01 14:57:14.111551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:14.111390 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk" event={"ID":"9e89169b-766a-413b-a844-bafa803a4911","Type":"ContainerDied","Data":"56411f6f8f03fd7a004a4b596873815f7a9b64c8aa53c74115b9d76c5852d6be"} Nov 01 14:57:14.111551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:14.111435 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56411f6f8f03fd7a004a4b596873815f7a9b64c8aa53c74115b9d76c5852d6be" Nov 01 14:57:15.009452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:15.009417 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:15.470820 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:15.470785 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:16.268256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:16.268194 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:16.434015 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:16.433987 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:18.540401 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:18.540370 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:20.971101 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.971061 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk"] Nov 01 14:57:20.971911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.971894 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9e89169b-766a-413b-a844-bafa803a4911" containerName="revoke-keys" Nov 01 14:57:20.971958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.971913 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e89169b-766a-413b-a844-bafa803a4911" containerName="revoke-keys" Nov 01 14:57:20.971958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.971943 5473 cpu_manager.go:414] "containerMap: removing container" podUID="015329eb-fabc-4993-a9f1-62f31728cc4f" containerName="revoke-keys" Nov 01 14:57:20.971958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.971951 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015329eb-fabc-4993-a9f1-62f31728cc4f" containerName="revoke-keys" Nov 01 14:57:20.972113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.972102 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e89169b-766a-413b-a844-bafa803a4911" containerName="revoke-keys" Nov 01 14:57:20.972152 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.972119 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="015329eb-fabc-4993-a9f1-62f31728cc4f" containerName="revoke-keys" Nov 01 14:57:20.976339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:20.976322 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.029628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.029597 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk"] Nov 01 14:57:21.058350 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.058320 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6czc\" (UniqueName: \"kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.058533 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.058394 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.058592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.058529 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.073195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.073160 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:57:21.073371 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.073272 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:57:21.160000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.159963 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6czc\" (UniqueName: \"kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.160188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.160043 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.160188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.160172 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.161039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.161010 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.163268 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.163207 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.217682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.217654 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6czc\" (UniqueName: \"kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc\") pod \"maas-api-revoke-sub-f0fb20cd-trzdk\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.261446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.261384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:21.290010 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.289987 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:21.458239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.458191 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7"] Nov 01 14:57:21.468679 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.467678 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521539 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521594 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521626 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521662 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521758 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521799 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxlt\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521835 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521881 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.528469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.521919 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.529316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.529237 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:21.534411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.532464 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl"] Nov 01 14:57:21.538684 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.537635 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.554899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.551448 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7"] Nov 01 14:57:21.562055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.562015 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl"] Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623549 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgm5\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623601 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623653 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623695 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623727 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623761 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623790 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623837 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623883 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623924 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623953 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.623995 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624021 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624064 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624109 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624199 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.624333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624283 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.625554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624453 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxlt\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.625554 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.624683 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.625690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.625567 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.625690 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.625580 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.625813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.625781 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.626166 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.626139 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.628408 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.627876 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.628862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.628838 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.682395 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.682371 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxlt\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.683022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.682998 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.684339 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:21.684318 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9e7705_837f_4db5_8155_3d68b578895e.slice/crio-ca7266c2ff38fcc8cae2ebddb3991829f090dc6ff60aafe4a114892bfdbfa2b1 WatchSource:0}: Error finding container ca7266c2ff38fcc8cae2ebddb3991829f090dc6ff60aafe4a114892bfdbfa2b1: Status 404 returned error can't find the container with id ca7266c2ff38fcc8cae2ebddb3991829f090dc6ff60aafe4a114892bfdbfa2b1 Nov 01 14:57:21.687163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.687143 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk"] Nov 01 14:57:21.725815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.725783 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.725932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.725840 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.725932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.725872 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.725932 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.725917 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.725974 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726010 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726115 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726078 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726162 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726273 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgm5\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726476 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726488 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.726747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.726706 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.727111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.727080 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.727247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.727109 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.729161 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.729138 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.729305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.729286 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.783260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.783177 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.783599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.783575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgm5\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5\") pod \"e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.830038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.830013 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:21.877459 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.877428 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:21.970308 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.970159 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g"] Nov 01 14:57:21.976963 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:21.976938 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.030445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.025931 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g"] Nov 01 14:57:22.037496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.035944 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.037496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.036118 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.037496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.036296 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdg7\" (UniqueName: \"kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.142241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.135701 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:22.147841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.145896 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.147841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.145986 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdg7\" (UniqueName: \"kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.147841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.146236 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.147841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.146984 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.150163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.150141 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.184195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.184166 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" event={"ID":"4e9e7705-837f-4db5-8155-3d68b578895e","Type":"ContainerStarted","Data":"f0b99e385abed0c471235d043c5cfeec25b5b4201fe2428d0531c4ef409ceec7"} Nov 01 14:57:22.184423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.184402 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" event={"ID":"4e9e7705-837f-4db5-8155-3d68b578895e","Type":"ContainerStarted","Data":"ca7266c2ff38fcc8cae2ebddb3991829f090dc6ff60aafe4a114892bfdbfa2b1"} Nov 01 14:57:22.219798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.219207 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdg7\" (UniqueName: \"kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7\") pod \"maas-api-revoke-sub-0c72192e-xcr9g\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.247243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.243020 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7"] Nov 01 14:57:22.266243 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:22.264364 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641a2393_367a_44ab_a86f_7b44316c0d9c.slice/crio-c605186069988f7943ab9adc976f34652f6b1b361c24db8e66ae3773348c663b WatchSource:0}: Error finding container c605186069988f7943ab9adc976f34652f6b1b361c24db8e66ae3773348c663b: Status 404 returned error can't find the container with id c605186069988f7943ab9adc976f34652f6b1b361c24db8e66ae3773348c663b Nov 01 14:57:22.268004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.267964 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:22.268201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.268183 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:22.268363 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.268347 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:22.318272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.317305 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl"] Nov 01 14:57:22.318466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.318443 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:22.321627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.321596 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:22.321734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.321673 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:22.321734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.321718 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:22.323508 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.323462 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" podStartSLOduration=2.3234481860000002 podStartE2EDuration="2.323448186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:22.323285651 +0000 UTC m=+2381.381437501" watchObservedRunningTime="2026-11-01 14:57:22.323448186 +0000 UTC m=+2381.381600036" Nov 01 14:57:22.343682 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.343599 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:22.809944 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:22.809917 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g"] Nov 01 14:57:23.194272 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.194204 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" event={"ID":"641a2393-367a-44ab-a86f-7b44316c0d9c","Type":"ContainerStarted","Data":"0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089"} Nov 01 14:57:23.194701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.194282 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" event={"ID":"641a2393-367a-44ab-a86f-7b44316c0d9c","Type":"ContainerStarted","Data":"c605186069988f7943ab9adc976f34652f6b1b361c24db8e66ae3773348c663b"} Nov 01 14:57:23.196333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.196301 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" event={"ID":"fb918a81-0960-4265-baf2-285161272836","Type":"ContainerStarted","Data":"daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c"} Nov 01 14:57:23.196484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.196358 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" event={"ID":"fb918a81-0960-4265-baf2-285161272836","Type":"ContainerStarted","Data":"7edf8342b8a96be51962e8e6263263e5b73dbdaddcba5917b9d253958002bf40"} Nov 01 14:57:23.198082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.198055 5473 generic.go:356] "Generic (PLEG): container finished" podID="4e9e7705-837f-4db5-8155-3d68b578895e" containerID="f0b99e385abed0c471235d043c5cfeec25b5b4201fe2428d0531c4ef409ceec7" exitCode=0 Nov 01 14:57:23.198184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.198130 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" event={"ID":"4e9e7705-837f-4db5-8155-3d68b578895e","Type":"ContainerDied","Data":"f0b99e385abed0c471235d043c5cfeec25b5b4201fe2428d0531c4ef409ceec7"} Nov 01 14:57:23.199577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.199558 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" event={"ID":"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf","Type":"ContainerStarted","Data":"f55b5530353159d28eb3dc68f58ed459d6c78b23c5027af59ee44bf7b5c6d17c"} Nov 01 14:57:23.321898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.321842 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podStartSLOduration=2.321829395 podStartE2EDuration="2.321829395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:23.318618772 +0000 UTC m=+2382.376770622" watchObservedRunningTime="2026-11-01 14:57:23.321829395 +0000 UTC m=+2382.379981245" Nov 01 14:57:23.606585 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.606533 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podStartSLOduration=2.606517517 podStartE2EDuration="2.606517517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:23.564540138 +0000 UTC m=+2382.622691989" watchObservedRunningTime="2026-11-01 14:57:23.606517517 +0000 UTC m=+2382.664669368" Nov 01 14:57:23.609557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.609533 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s"] Nov 01 14:57:23.613651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.613636 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.681176 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.681139 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s"] Nov 01 14:57:23.709158 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.709127 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.709366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.709168 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.709366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.709281 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8flj\" (UniqueName: \"kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.810326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.810288 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.810511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.810337 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.810511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.810492 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8flj\" (UniqueName: \"kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.811052 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.811019 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.813493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.813464 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.841376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.841291 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:23.875887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.861879 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 14:57:23.875887 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.875440 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 14:57:23.878432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.878406 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:23.882469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.882357 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 14:57:23.882469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.882406 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 14:57:23.884692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.884668 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8flj\" (UniqueName: \"kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj\") pod \"maas-api-revoke-sub-424717e3-6mp8s\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:23.946241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:23.941742 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:24.210008 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.209983 5473 generic.go:356] "Generic (PLEG): container finished" podID="0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" containerID="b3ace4dbfe6c4314945b21f337078cd64a761959673e1a0f66ca0aebc58e81f3" exitCode=0 Nov 01 14:57:24.210632 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.210601 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" event={"ID":"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf","Type":"ContainerDied","Data":"b3ace4dbfe6c4314945b21f337078cd64a761959673e1a0f66ca0aebc58e81f3"} Nov 01 14:57:24.486733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.486457 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s"] Nov 01 14:57:24.487764 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:24.487736 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bb504f_6d27_47a9_96fe_671da3f09f4b.slice/crio-55a74b209f08f54302dd29accc95e7e792e688d359f7e5b8ad2f613042c90648 WatchSource:0}: Error finding container 55a74b209f08f54302dd29accc95e7e792e688d359f7e5b8ad2f613042c90648: Status 404 returned error can't find the container with id 55a74b209f08f54302dd29accc95e7e792e688d359f7e5b8ad2f613042c90648 Nov 01 14:57:24.591145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.591118 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:24.622660 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.622634 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca\") pod \"4e9e7705-837f-4db5-8155-3d68b578895e\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " Nov 01 14:57:24.622772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.622679 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca\") pod \"4e9e7705-837f-4db5-8155-3d68b578895e\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " Nov 01 14:57:24.622814 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.622797 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc\" (UniqueName: \"kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc\") pod \"4e9e7705-837f-4db5-8155-3d68b578895e\" (UID: \"4e9e7705-837f-4db5-8155-3d68b578895e\") " Nov 01 14:57:24.623156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.623135 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca" pod "4e9e7705-837f-4db5-8155-3d68b578895e" (UID: "4e9e7705-837f-4db5-8155-3d68b578895e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:57:24.647529 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.647362 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc" pod "4e9e7705-837f-4db5-8155-3d68b578895e" (UID: "4e9e7705-837f-4db5-8155-3d68b578895e"). InnerVolumeSpecName "kube-api-access-f6czc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:57:24.673381 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.673346 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca" pod "4e9e7705-837f-4db5-8155-3d68b578895e" (UID: "4e9e7705-837f-4db5-8155-3d68b578895e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:57:24.751362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.724950 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6czc\" (UniqueName: \"kubernetes.io/projected/4e9e7705-837f-4db5-8155-3d68b578895e-kube-api-access-f6czc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:24.751603 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.751579 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:24.778383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.778335 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9e7705-837f-4db5-8155-3d68b578895e-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:24.832631 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.832600 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 14:57:24.836316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.836279 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 14:57:24.879867 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.879834 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 14:57:24.880133 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:24.880105 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 14:57:25.218596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.218569 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" Nov 01 14:57:25.219000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.218610 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk" event={"ID":"4e9e7705-837f-4db5-8155-3d68b578895e","Type":"ContainerDied","Data":"ca7266c2ff38fcc8cae2ebddb3991829f090dc6ff60aafe4a114892bfdbfa2b1"} Nov 01 14:57:25.219000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.218656 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7266c2ff38fcc8cae2ebddb3991829f090dc6ff60aafe4a114892bfdbfa2b1" Nov 01 14:57:25.221018 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.220985 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" event={"ID":"44bb504f-6d27-47a9-96fe-671da3f09f4b","Type":"ContainerStarted","Data":"9d8dfbf7534a4ef7a7a222c737565cad97550ef5e39521a1a364188005290831"} Nov 01 14:57:25.221167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.221039 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" event={"ID":"44bb504f-6d27-47a9-96fe-671da3f09f4b","Type":"ContainerStarted","Data":"55a74b209f08f54302dd29accc95e7e792e688d359f7e5b8ad2f613042c90648"} Nov 01 14:57:25.340375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.339948 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" podStartSLOduration=2.339929224 podStartE2EDuration="2.339929224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:25.338607494 +0000 UTC m=+2384.396759345" watchObservedRunningTime="2026-11-01 14:57:25.339929224 +0000 UTC m=+2384.398081090" Nov 01 14:57:25.515607 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.515370 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:25.695822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.695777 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7\" (UniqueName: \"kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7\") pod \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " Nov 01 14:57:25.696001 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.695881 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca\") pod \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " Nov 01 14:57:25.696044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.696001 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca\") pod \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\" (UID: \"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf\") " Nov 01 14:57:25.697016 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.696988 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca" pod "0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" (UID: "0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:57:25.698920 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.698889 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca" pod "0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" (UID: "0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:57:25.699086 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.699061 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7" pod "0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" (UID: "0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf"). InnerVolumeSpecName "kube-api-access-5qdg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:57:25.798206 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.798179 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qdg7\" (UniqueName: \"kubernetes.io/projected/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-kube-api-access-5qdg7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:25.798326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.798208 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:25.798326 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.798235 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:25.835587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.835536 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 14:57:25.858870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.858788 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 14:57:25.878794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.878697 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 14:57:25.878794 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:25.878749 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 14:57:26.264563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.264527 5473 generic.go:356] "Generic (PLEG): container finished" podID="44bb504f-6d27-47a9-96fe-671da3f09f4b" containerID="9d8dfbf7534a4ef7a7a222c737565cad97550ef5e39521a1a364188005290831" exitCode=0 Nov 01 14:57:26.272652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.265153 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" event={"ID":"44bb504f-6d27-47a9-96fe-671da3f09f4b","Type":"ContainerDied","Data":"9d8dfbf7534a4ef7a7a222c737565cad97550ef5e39521a1a364188005290831"} Nov 01 14:57:26.275237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.274614 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" event={"ID":"0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf","Type":"ContainerDied","Data":"f55b5530353159d28eb3dc68f58ed459d6c78b23c5027af59ee44bf7b5c6d17c"} Nov 01 14:57:26.275237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.274657 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55b5530353159d28eb3dc68f58ed459d6c78b23c5027af59ee44bf7b5c6d17c" Nov 01 14:57:26.275237 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.274708 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g" Nov 01 14:57:26.371239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.368801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:26.832647 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.832605 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 14:57:26.832964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.832932 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 14:57:26.878676 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.878639 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 14:57:26.878853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:26.878691 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 14:57:27.517321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.517234 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:27.622255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.622143 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca\") pod \"44bb504f-6d27-47a9-96fe-671da3f09f4b\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " Nov 01 14:57:27.622440 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.622304 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj\" (UniqueName: \"kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj\") pod \"44bb504f-6d27-47a9-96fe-671da3f09f4b\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " Nov 01 14:57:27.622440 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.622363 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca\") pod \"44bb504f-6d27-47a9-96fe-671da3f09f4b\" (UID: \"44bb504f-6d27-47a9-96fe-671da3f09f4b\") " Nov 01 14:57:27.622888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.622861 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca" pod "44bb504f-6d27-47a9-96fe-671da3f09f4b" (UID: "44bb504f-6d27-47a9-96fe-671da3f09f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:57:27.647241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.632262 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca" pod "44bb504f-6d27-47a9-96fe-671da3f09f4b" (UID: "44bb504f-6d27-47a9-96fe-671da3f09f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:57:27.647241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.641430 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj" pod "44bb504f-6d27-47a9-96fe-671da3f09f4b" (UID: "44bb504f-6d27-47a9-96fe-671da3f09f4b"). InnerVolumeSpecName "kube-api-access-k8flj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:57:27.723514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.723483 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8flj\" (UniqueName: \"kubernetes.io/projected/44bb504f-6d27-47a9-96fe-671da3f09f4b-kube-api-access-k8flj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:27.723514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.723512 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:27.723710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.723523 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44bb504f-6d27-47a9-96fe-671da3f09f4b-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:27.832240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.831547 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 14:57:27.832240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.831618 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 14:57:27.878953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.878867 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 14:57:27.878953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:27.878922 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 14:57:28.107127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.107088 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:28.292387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.292363 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" Nov 01 14:57:28.292630 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.292396 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s" event={"ID":"44bb504f-6d27-47a9-96fe-671da3f09f4b","Type":"ContainerDied","Data":"55a74b209f08f54302dd29accc95e7e792e688d359f7e5b8ad2f613042c90648"} Nov 01 14:57:28.292805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.292792 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55a74b209f08f54302dd29accc95e7e792e688d359f7e5b8ad2f613042c90648" Nov 01 14:57:28.830513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.830467 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 14:57:28.830927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.830548 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 14:57:28.878562 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.878528 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 01 14:57:28.878752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:28.878579 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 01 14:57:29.404482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:29.404444 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:29.820575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:29.820535 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:29.945170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:29.945135 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:29.946036 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:29.946015 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:29.946931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:29.946906 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 14:57:30.125344 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:30.125269 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:30.125666 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:30.125648 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:30.126490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:30.126471 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 14:57:31.108353 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:57:31.108312 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:57:31.307113 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:31.307075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:33.592789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:33.592759 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:37.207546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:37.207508 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:38.577186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577152 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk"] Nov 01 14:57:38.577750 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577734 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" containerName="revoke-keys" Nov 01 14:57:38.577803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577751 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" containerName="revoke-keys" Nov 01 14:57:38.577803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577783 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4e9e7705-837f-4db5-8155-3d68b578895e" containerName="revoke-keys" Nov 01 14:57:38.577803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577788 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9e7705-837f-4db5-8155-3d68b578895e" containerName="revoke-keys" Nov 01 14:57:38.577894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577808 5473 cpu_manager.go:414] "containerMap: removing container" podUID="44bb504f-6d27-47a9-96fe-671da3f09f4b" containerName="revoke-keys" Nov 01 14:57:38.577894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577815 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44bb504f-6d27-47a9-96fe-671da3f09f4b" containerName="revoke-keys" Nov 01 14:57:38.578005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577900 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9e7705-837f-4db5-8155-3d68b578895e" containerName="revoke-keys" Nov 01 14:57:38.578005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577913 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="44bb504f-6d27-47a9-96fe-671da3f09f4b" containerName="revoke-keys" Nov 01 14:57:38.578005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.577921 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" containerName="revoke-keys" Nov 01 14:57:38.581369 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.581349 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.634541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.634517 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk"] Nov 01 14:57:38.679157 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.679130 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmxl\" (UniqueName: \"kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.679307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.679162 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.679307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.679273 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.680386 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.680359 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:57:38.680491 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.680397 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:57:38.780964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.780940 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmxl\" (UniqueName: \"kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.781085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.780973 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.781085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.781074 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.781719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.781699 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.783288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.783269 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.843482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.843427 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmxl\" (UniqueName: \"kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl\") pod \"maas-api-revoke-sub-204d04cb-5kkpk\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:38.894674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:38.894645 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:39.266648 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:39.266623 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8044957f_49e0_4ffe_aa43_6133e6f16629.slice/crio-0e9501313de35964ee6cba792bc716536bc91e1a805d6e9df3b7b1f7930dea7e WatchSource:0}: Error finding container 0e9501313de35964ee6cba792bc716536bc91e1a805d6e9df3b7b1f7930dea7e: Status 404 returned error can't find the container with id 0e9501313de35964ee6cba792bc716536bc91e1a805d6e9df3b7b1f7930dea7e Nov 01 14:57:39.271130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:39.271101 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk"] Nov 01 14:57:39.372393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:39.372361 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" event={"ID":"8044957f-49e0-4ffe-aa43-6133e6f16629","Type":"ContainerStarted","Data":"0e9501313de35964ee6cba792bc716536bc91e1a805d6e9df3b7b1f7930dea7e"} Nov 01 14:57:40.244081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.244052 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5"] Nov 01 14:57:40.248293 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.248275 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.294128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.294101 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp"] Nov 01 14:57:40.298073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.298057 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.300809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.300784 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.300915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.300891 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.300968 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.300949 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.301028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.300979 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4lh\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.301097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.301077 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.301144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.301107 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.301144 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.301128 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.301274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.301260 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.301330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.301301 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.306608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.306585 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-007270-openshift-default-dockercfg-78vkt\"" Nov 01 14:57:40.346713 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.346636 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5"] Nov 01 14:57:40.354282 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.354259 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp"] Nov 01 14:57:40.382250 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.382173 5473 generic.go:356] "Generic (PLEG): container finished" podID="8044957f-49e0-4ffe-aa43-6133e6f16629" containerID="9c583febbcdc9b3ad0d6ef9fda3c68abcecce0cf6a41ada411598f13ac171f1c" exitCode=0 Nov 01 14:57:40.382411 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.382260 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" event={"ID":"8044957f-49e0-4ffe-aa43-6133e6f16629","Type":"ContainerDied","Data":"9c583febbcdc9b3ad0d6ef9fda3c68abcecce0cf6a41ada411598f13ac171f1c"} Nov 01 14:57:40.403556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403528 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.403686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403571 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.403745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403709 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.403745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403740 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.403878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403772 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.403878 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403797 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.403982 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404007 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404032 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404100 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404148 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404244 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffv98\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404286 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4lh\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404319 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404336 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404445 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404489 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404481 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404513 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404546 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.404699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404577 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404870 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.404970 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.404948 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.405020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.405005 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.406544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.406524 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.406718 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.406700 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.460505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.460484 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.460815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.460793 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4lh\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.506263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506212 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506275 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506325 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506379 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506414 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffv98\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506448 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506525 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506533 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506625 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506570 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506624 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506645 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.506823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506796 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.507000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.506972 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.507129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.507107 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.508640 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.508620 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.508864 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.508846 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.561020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.560995 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:40.562774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.562754 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.562869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.562850 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffv98\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.614444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.614413 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:40.982247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.981553 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5"] Nov 01 14:57:40.987208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.987154 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:40.987339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.987261 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:40.987339 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:40.987306 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:41.051570 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.051534 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp"] Nov 01 14:57:41.068459 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:57:41.068432 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0c442d_f399_48f1_8bd6_9c38e916fd39.slice/crio-108038e8a019f412b407c8208b02d536fe5e1c4c12d34b5b45dbcaad15d8ddbd WatchSource:0}: Error finding container 108038e8a019f412b407c8208b02d536fe5e1c4c12d34b5b45dbcaad15d8ddbd: Status 404 returned error can't find the container with id 108038e8a019f412b407c8208b02d536fe5e1c4c12d34b5b45dbcaad15d8ddbd Nov 01 14:57:41.071043 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.070988 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:41.071168 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.071113 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:41.071288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.071173 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:57:41.390707 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.390673 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" event={"ID":"ae0c442d-f399-48f1-8bd6-9c38e916fd39","Type":"ContainerStarted","Data":"47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c"} Nov 01 14:57:41.391238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.390762 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" event={"ID":"ae0c442d-f399-48f1-8bd6-9c38e916fd39","Type":"ContainerStarted","Data":"108038e8a019f412b407c8208b02d536fe5e1c4c12d34b5b45dbcaad15d8ddbd"} Nov 01 14:57:41.392435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.392406 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" event={"ID":"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97","Type":"ContainerStarted","Data":"915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a"} Nov 01 14:57:41.392556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.392460 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" event={"ID":"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97","Type":"ContainerStarted","Data":"87fbb7d4b72cf1c8f879fab809cc7dcfcb2d4686242aeb654712ce017f442bf8"} Nov 01 14:57:41.613831 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.595798 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:41.616985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.616941 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:41.621664 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.621641 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:41.672000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.671877 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podStartSLOduration=1.671864172 podStartE2EDuration="1.671864172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:41.529144492 +0000 UTC m=+2400.587296344" watchObservedRunningTime="2026-11-01 14:57:41.671864172 +0000 UTC m=+2400.730016025" Nov 01 14:57:41.673327 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.673285 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podStartSLOduration=1.67327388 podStartE2EDuration="1.67327388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:57:41.668331449 +0000 UTC m=+2400.726483299" watchObservedRunningTime="2026-11-01 14:57:41.67327388 +0000 UTC m=+2400.731425730" Nov 01 14:57:41.746821 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.746782 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:41.755934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.755891 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca\") pod \"8044957f-49e0-4ffe-aa43-6133e6f16629\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " Nov 01 14:57:41.756423 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.756397 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca\") pod \"8044957f-49e0-4ffe-aa43-6133e6f16629\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " Nov 01 14:57:41.756674 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.756657 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl\" (UniqueName: \"kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl\") pod \"8044957f-49e0-4ffe-aa43-6133e6f16629\" (UID: \"8044957f-49e0-4ffe-aa43-6133e6f16629\") " Nov 01 14:57:41.770284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.765414 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca" pod "8044957f-49e0-4ffe-aa43-6133e6f16629" (UID: "8044957f-49e0-4ffe-aa43-6133e6f16629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:57:41.770284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.765635 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl" pod "8044957f-49e0-4ffe-aa43-6133e6f16629" (UID: "8044957f-49e0-4ffe-aa43-6133e6f16629"). InnerVolumeSpecName "kube-api-access-hqmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:57:41.770284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.767599 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca" pod "8044957f-49e0-4ffe-aa43-6133e6f16629" (UID: "8044957f-49e0-4ffe-aa43-6133e6f16629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:57:41.869241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.868385 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:41.869241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.868429 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8044957f-49e0-4ffe-aa43-6133e6f16629-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:41.869241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:41.868446 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmxl\" (UniqueName: \"kubernetes.io/projected/8044957f-49e0-4ffe-aa43-6133e6f16629-kube-api-access-hqmxl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:57:42.403993 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.403952 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" Nov 01 14:57:42.404501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.403990 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk" event={"ID":"8044957f-49e0-4ffe-aa43-6133e6f16629","Type":"ContainerDied","Data":"0e9501313de35964ee6cba792bc716536bc91e1a805d6e9df3b7b1f7930dea7e"} Nov 01 14:57:42.404501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.404052 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e9501313de35964ee6cba792bc716536bc91e1a805d6e9df3b7b1f7930dea7e" Nov 01 14:57:42.569512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.568281 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 01 14:57:42.569512 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.568341 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 01 14:57:42.616024 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.615982 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 14:57:42.616192 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:42.616052 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 14:57:43.563242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:43.562230 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 01 14:57:43.563242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:43.562291 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 01 14:57:43.623248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:43.621565 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 14:57:43.623248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:43.621627 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 14:57:43.914659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:43.914581 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:57:43.917659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:43.917638 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 14:57:44.562648 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:44.562606 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 01 14:57:44.562975 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:44.562946 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 01 14:57:44.615078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:44.615050 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 14:57:44.615466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:44.615103 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 14:57:44.863319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:44.863206 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:45.562239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:45.561633 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 01 14:57:45.562239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:45.561698 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 01 14:57:45.615676 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:45.615642 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 14:57:45.616072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:45.615703 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 14:57:46.222748 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:46.222665 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:46.565239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:46.564608 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 01 14:57:46.565239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:46.564671 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 01 14:57:46.614899 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:46.614871 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 14:57:46.615039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:46.614918 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 14:57:47.562134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:47.562099 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Nov 01 14:57:47.562579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:47.562157 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Nov 01 14:57:47.614936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:47.614900 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 01 14:57:47.615119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:47.614955 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 01 14:57:48.707769 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:48.707729 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:48.708918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:48.708890 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:48.709743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:48.709712 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 14:57:48.867858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:48.867821 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:48.868195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:48.868170 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:48.869579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:48.869557 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 14:57:49.889140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:49.889101 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:50.104941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:50.104908 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:51.711619 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:51.711588 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:54.433739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:54.433706 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:54.610701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:54.610670 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:55.769804 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:55.769766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:55.943325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:55.943279 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:56.368235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:56.368183 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw"] Nov 01 14:57:56.368428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:56.368407 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" containerID="cri-o://2585f969f3111591bf5ecb55379254a681ad85ba5103bf7e676eb993b4173472" gracePeriod=30 Nov 01 14:57:56.417716 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:56.417683 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:57:56.595680 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:57:56.595625 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:00.296057 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.296025 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw"] Nov 01 14:58:00.296610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.296596 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8044957f-49e0-4ffe-aa43-6133e6f16629" containerName="revoke-keys" Nov 01 14:58:00.296610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.296610 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8044957f-49e0-4ffe-aa43-6133e6f16629" containerName="revoke-keys" Nov 01 14:58:00.296732 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.296723 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8044957f-49e0-4ffe-aa43-6133e6f16629" containerName="revoke-keys" Nov 01 14:58:00.301772 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.301750 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.353201 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.353160 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw"] Nov 01 14:58:00.398041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.398013 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:58:00.398191 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.398134 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:58:00.404473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.404432 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.404695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.404678 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kjs\" (UniqueName: \"kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.404765 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.404707 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.505774 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.505739 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kjs\" (UniqueName: \"kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.505946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.505780 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.505946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.505897 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.506692 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.506664 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.508269 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.508247 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.562351 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.562301 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kjs\" (UniqueName: \"kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs\") pod \"maas-api-revoke-sub-10cd46e2-nk5cw\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.615375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.615351 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:00.985021 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:00.984991 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5027c9ef_a0b0_49a8_af14_dcd1390e3e00.slice/crio-892c0cbad6637c66ae47d1169fc551b8341768e06099368594da45fd2599c1f6 WatchSource:0}: Error finding container 892c0cbad6637c66ae47d1169fc551b8341768e06099368594da45fd2599c1f6: Status 404 returned error can't find the container with id 892c0cbad6637c66ae47d1169fc551b8341768e06099368594da45fd2599c1f6 Nov 01 14:58:00.988971 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:00.988946 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw"] Nov 01 14:58:01.107678 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:58:01.107646 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:58:01.123121 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.123087 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:01.632243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.632196 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" event={"ID":"5027c9ef-a0b0-49a8-af14-dcd1390e3e00","Type":"ContainerStarted","Data":"c6da71600de5ce3e5431d655bf5f42f7c45085a489f9df4e86141bf851717350"} Nov 01 14:58:01.632670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.632275 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" event={"ID":"5027c9ef-a0b0-49a8-af14-dcd1390e3e00","Type":"ContainerStarted","Data":"892c0cbad6637c66ae47d1169fc551b8341768e06099368594da45fd2599c1f6"} Nov 01 14:58:01.634284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.634262 5473 generic.go:356] "Generic (PLEG): container finished" podID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerID="2585f969f3111591bf5ecb55379254a681ad85ba5103bf7e676eb993b4173472" exitCode=0 Nov 01 14:58:01.634402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.634324 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" event={"ID":"7d536e10-e370-47e1-a3ed-122c1b806f5b","Type":"ContainerDied","Data":"2585f969f3111591bf5ecb55379254a681ad85ba5103bf7e676eb993b4173472"} Nov 01 14:58:01.634402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.634376 5473 scope.go:122] "RemoveContainer" containerID="25d032ee285c78272d152a9c19fa08bb38e1f5c3ce1ace6eeacc5e57330da4e3" Nov 01 14:58:01.654469 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.654449 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:58:01.749579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.749494 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" podStartSLOduration=1.749481925 podStartE2EDuration="1.749481925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:01.746683908 +0000 UTC m=+2420.804835757" watchObservedRunningTime="2026-11-01 14:58:01.749481925 +0000 UTC m=+2420.807633783" Nov 01 14:58:01.824296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824253 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824382 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824422 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824557 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824522 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:01.824623 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824577 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824625 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824686 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824670 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824742 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824800 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824779 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824800 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert\") pod \"7d536e10-e370-47e1-a3ed-122c1b806f5b\" (UID: \"7d536e10-e370-47e1-a3ed-122c1b806f5b\") " Nov 01 14:58:01.824980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.824957 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:01.825575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.825432 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:01.825575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.825450 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:01.825575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.825512 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.825575 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.825537 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.825865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.825772 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:01.826873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.826848 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:01.827022 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.826882 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "kube-api-access-cjhn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:01.827116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.827096 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:01.827569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.827552 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:58:01.827688 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.827674 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy" pod "7d536e10-e370-47e1-a3ed-122c1b806f5b" (UID: "7d536e10-e370-47e1-a3ed-122c1b806f5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:01.927126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927096 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhn5\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-kube-api-access-cjhn5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.927126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927122 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.927126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927131 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.927378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927141 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.927378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927155 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.927378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927163 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d536e10-e370-47e1-a3ed-122c1b806f5b-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:01.927378 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:01.927173 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d536e10-e370-47e1-a3ed-122c1b806f5b-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:02.642455 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.642428 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" Nov 01 14:58:02.642896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.642475 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw" event={"ID":"7d536e10-e370-47e1-a3ed-122c1b806f5b","Type":"ContainerDied","Data":"5a6a8cd838d1f9571a6a38d8ee80f0aa3fccbd5ecde229359bdb34b0d0c21849"} Nov 01 14:58:02.642896 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.642530 5473 scope.go:122] "RemoveContainer" containerID="2585f969f3111591bf5ecb55379254a681ad85ba5103bf7e676eb993b4173472" Nov 01 14:58:02.644334 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.644309 5473 generic.go:356] "Generic (PLEG): container finished" podID="5027c9ef-a0b0-49a8-af14-dcd1390e3e00" containerID="c6da71600de5ce3e5431d655bf5f42f7c45085a489f9df4e86141bf851717350" exitCode=0 Nov 01 14:58:02.644441 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.644344 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" event={"ID":"5027c9ef-a0b0-49a8-af14-dcd1390e3e00","Type":"ContainerDied","Data":"c6da71600de5ce3e5431d655bf5f42f7c45085a489f9df4e86141bf851717350"} Nov 01 14:58:02.927354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.927247 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw"] Nov 01 14:58:02.935662 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:02.935634 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a90080-openshift-default-5df578f89-wm5hw"] Nov 01 14:58:03.084424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.084388 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" path="/var/lib/kubelet/pods/7d536e10-e370-47e1-a3ed-122c1b806f5b/volumes" Nov 01 14:58:03.805656 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.805635 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:03.950266 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.950168 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs\" (UniqueName: \"kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs\") pod \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " Nov 01 14:58:03.950417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.950366 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca\") pod \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " Nov 01 14:58:03.950483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.950467 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca\") pod \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\" (UID: \"5027c9ef-a0b0-49a8-af14-dcd1390e3e00\") " Nov 01 14:58:03.950763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.950740 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca" pod "5027c9ef-a0b0-49a8-af14-dcd1390e3e00" (UID: "5027c9ef-a0b0-49a8-af14-dcd1390e3e00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:03.950958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.950940 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:03.952431 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.952403 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs" pod "5027c9ef-a0b0-49a8-af14-dcd1390e3e00" (UID: "5027c9ef-a0b0-49a8-af14-dcd1390e3e00"). InnerVolumeSpecName "kube-api-access-l4kjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:03.952939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:03.952908 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca" pod "5027c9ef-a0b0-49a8-af14-dcd1390e3e00" (UID: "5027c9ef-a0b0-49a8-af14-dcd1390e3e00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:04.051985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:04.051948 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:04.051985 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:04.051980 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kjs\" (UniqueName: \"kubernetes.io/projected/5027c9ef-a0b0-49a8-af14-dcd1390e3e00-kube-api-access-l4kjs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:04.661551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:04.661520 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" Nov 01 14:58:04.661551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:04.661533 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw" event={"ID":"5027c9ef-a0b0-49a8-af14-dcd1390e3e00","Type":"ContainerDied","Data":"892c0cbad6637c66ae47d1169fc551b8341768e06099368594da45fd2599c1f6"} Nov 01 14:58:04.661767 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:04.661584 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="892c0cbad6637c66ae47d1169fc551b8341768e06099368594da45fd2599c1f6" Nov 01 14:58:05.608657 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:05.608622 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:07.757047 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:07.757012 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74"] Nov 01 14:58:07.864722 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:07.864689 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk"] Nov 01 14:58:07.909577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:07.909546 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:08.135005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:08.134963 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:08.307934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:08.307900 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:09.633081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:09.633042 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:10.128939 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:10.128905 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:10.294851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:10.294820 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:11.827241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:11.826708 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:12.758316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:12.758276 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" podUID="ef1f9291-f31d-4a6c-9bab-25004dd21fee" containerName="payload-processing" containerID="cri-o://95e227edf2f01087a14c46c61ed42d1daa6b61b4c393130c4842ccb607feb51e" gracePeriod=25 Nov 01 14:58:12.865537 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:12.865484 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" podUID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" containerName="payload-pre-processing" containerID="cri-o://341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486" gracePeriod=25 Nov 01 14:58:12.981777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:12.981747 5473 generic.go:356] "Generic (PLEG): container finished" podID="ef1f9291-f31d-4a6c-9bab-25004dd21fee" containerID="95e227edf2f01087a14c46c61ed42d1daa6b61b4c393130c4842ccb607feb51e" exitCode=0 Nov 01 14:58:12.981886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:12.981832 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" event={"ID":"ef1f9291-f31d-4a6c-9bab-25004dd21fee","Type":"ContainerDied","Data":"95e227edf2f01087a14c46c61ed42d1daa6b61b4c393130c4842ccb607feb51e"} Nov 01 14:58:13.024067 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.024030 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:58:13.129785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.129758 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:58:13.133982 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.133962 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume\") pod \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " Nov 01 14:58:13.134085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.134062 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6\" (UniqueName: \"kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6\") pod \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\" (UID: \"ef1f9291-f31d-4a6c-9bab-25004dd21fee\") " Nov 01 14:58:13.134486 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.134449 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume" pod "ef1f9291-f31d-4a6c-9bab-25004dd21fee" (UID: "ef1f9291-f31d-4a6c-9bab-25004dd21fee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:13.134601 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.134570 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef1f9291-f31d-4a6c-9bab-25004dd21fee-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:13.136501 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.136472 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6" pod "ef1f9291-f31d-4a6c-9bab-25004dd21fee" (UID: "ef1f9291-f31d-4a6c-9bab-25004dd21fee"). InnerVolumeSpecName "kube-api-access-ggkb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:13.235505 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.235475 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g\" (UniqueName: \"kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g\") pod \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " Nov 01 14:58:13.235676 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.235659 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume\") pod \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\" (UID: \"e9ffa5cc-e50d-4829-97b0-85c0e826ecef\") " Nov 01 14:58:13.235976 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.235955 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggkb6\" (UniqueName: \"kubernetes.io/projected/ef1f9291-f31d-4a6c-9bab-25004dd21fee-kube-api-access-ggkb6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:13.236097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.235992 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume" pod "e9ffa5cc-e50d-4829-97b0-85c0e826ecef" (UID: "e9ffa5cc-e50d-4829-97b0-85c0e826ecef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:13.237547 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.237528 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g" pod "e9ffa5cc-e50d-4829-97b0-85c0e826ecef" (UID: "e9ffa5cc-e50d-4829-97b0-85c0e826ecef"). InnerVolumeSpecName "kube-api-access-7244g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:13.337030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.336957 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:13.337030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.336986 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7244g\" (UniqueName: \"kubernetes.io/projected/e9ffa5cc-e50d-4829-97b0-85c0e826ecef-kube-api-access-7244g\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:13.658578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.658496 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67"] Nov 01 14:58:13.659087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659072 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ef1f9291-f31d-4a6c-9bab-25004dd21fee" containerName="payload-processing" Nov 01 14:58:13.659135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659089 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1f9291-f31d-4a6c-9bab-25004dd21fee" containerName="payload-processing" Nov 01 14:58:13.659135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659102 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" Nov 01 14:58:13.659135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659107 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" Nov 01 14:58:13.659135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659123 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" containerName="payload-pre-processing" Nov 01 14:58:13.659135 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659128 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" containerName="payload-pre-processing" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659144 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5027c9ef-a0b0-49a8-af14-dcd1390e3e00" containerName="revoke-keys" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659149 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5027c9ef-a0b0-49a8-af14-dcd1390e3e00" containerName="revoke-keys" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659249 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1f9291-f31d-4a6c-9bab-25004dd21fee" containerName="payload-processing" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659258 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659266 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" containerName="payload-pre-processing" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659273 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5027c9ef-a0b0-49a8-af14-dcd1390e3e00" containerName="revoke-keys" Nov 01 14:58:13.659315 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.659280 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" Nov 01 14:58:13.662936 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.662915 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.716980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.716942 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67"] Nov 01 14:58:13.717511 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.717487 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:58:13.718167 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.718145 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:58:13.841328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.841296 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.841464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.841445 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl9vm\" (UniqueName: \"kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.841525 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.841510 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.872065 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.871997 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:13.942472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.942399 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl9vm\" (UniqueName: \"kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.942472 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.942430 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.942705 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.942564 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.943197 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.943172 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.944918 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.944893 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:13.991008 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.990969 5473 generic.go:356] "Generic (PLEG): container finished" podID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" containerID="341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486" exitCode=0 Nov 01 14:58:13.991162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.991033 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" Nov 01 14:58:13.991162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.991052 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" event={"ID":"e9ffa5cc-e50d-4829-97b0-85c0e826ecef","Type":"ContainerDied","Data":"341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486"} Nov 01 14:58:13.991162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.991106 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" event={"ID":"e9ffa5cc-e50d-4829-97b0-85c0e826ecef","Type":"ContainerDied","Data":"f9a0f217c52d59d56c85dcd91ffabf6299558728a473f6923086122709417f60"} Nov 01 14:58:13.991162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.991128 5473 scope.go:122] "RemoveContainer" containerID="341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486" Nov 01 14:58:13.993044 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.993023 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" Nov 01 14:58:13.993164 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:13.993023 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74" event={"ID":"ef1f9291-f31d-4a6c-9bab-25004dd21fee","Type":"ContainerDied","Data":"d326f50aca05696670cf591c9b6b4a5ae3702b14627d51c7f972fa8a6f6b6e37"} Nov 01 14:58:14.001708 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.001685 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl9vm\" (UniqueName: \"kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm\") pod \"maas-api-revoke-sub-bc8e885c-42s67\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:14.002573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.002558 5473 scope.go:122] "RemoveContainer" containerID="341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486" Nov 01 14:58:14.002891 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:58:14.002864 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486\": container with ID starting with 341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486 not found: ID does not exist" containerID="341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486" Nov 01 14:58:14.002978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.002900 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486"} err="failed to get container status \"341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486\": rpc error: code = NotFound desc = could not find container \"341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486\": container with ID starting with 341b32223673ffe8144b6f02e05e35723b3c1993c65118150e6233a3e9749486 not found: ID does not exist" Nov 01 14:58:14.002978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.002923 5473 scope.go:122] "RemoveContainer" containerID="95e227edf2f01087a14c46c61ed42d1daa6b61b4c393130c4842ccb607feb51e" Nov 01 14:58:14.102852 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.102801 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk" podUID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.185:50052\" within 1s: context deadline exceeded" Nov 01 14:58:14.102996 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:14.102880 5473 logging.go:55] [core] [Channel #6140 SubChannel #6141]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.185:50052", ServerName: "10.134.0.185:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.185:50052: operation was canceled" Nov 01 14:58:14.156580 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.156553 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk"] Nov 01 14:58:14.160318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.160294 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7197e2-5d5d5b9987-nw8dk"] Nov 01 14:58:14.268906 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.268880 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74"] Nov 01 14:58:14.272452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.272435 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7197e2-6cdcdf764c-xxm74"] Nov 01 14:58:14.276462 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.276444 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:14.644909 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:14.644877 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e500828_6d5d_4b42_816d_41aba06545b7.slice/crio-207b68fac784a2a5d6a5d93bd609174c64eb58c037bc950744ac3e4d290dbf63 WatchSource:0}: Error finding container 207b68fac784a2a5d6a5d93bd609174c64eb58c037bc950744ac3e4d290dbf63: Status 404 returned error can't find the container with id 207b68fac784a2a5d6a5d93bd609174c64eb58c037bc950744ac3e4d290dbf63 Nov 01 14:58:14.649542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.649527 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67"] Nov 01 14:58:14.974119 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:14.974084 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:15.002888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:15.002852 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" event={"ID":"7e500828-6d5d-4b42-816d-41aba06545b7","Type":"ContainerStarted","Data":"a0554e2cfbb3334a85faa9cdb4b2123959d20f82d5435d42923179ba6c535622"} Nov 01 14:58:15.003084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:15.002918 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" event={"ID":"7e500828-6d5d-4b42-816d-41aba06545b7","Type":"ContainerStarted","Data":"207b68fac784a2a5d6a5d93bd609174c64eb58c037bc950744ac3e4d290dbf63"} Nov 01 14:58:15.081798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:15.081771 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ffa5cc-e50d-4829-97b0-85c0e826ecef" path="/var/lib/kubelet/pods/e9ffa5cc-e50d-4829-97b0-85c0e826ecef/volumes" Nov 01 14:58:15.082194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:15.082181 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1f9291-f31d-4a6c-9bab-25004dd21fee" path="/var/lib/kubelet/pods/ef1f9291-f31d-4a6c-9bab-25004dd21fee/volumes" Nov 01 14:58:15.121509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:15.121454 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" podStartSLOduration=2.121439065 podStartE2EDuration="2.121439065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:15.118145755 +0000 UTC m=+2434.176297599" watchObservedRunningTime="2026-11-01 14:58:15.121439065 +0000 UTC m=+2434.179590909" Nov 01 14:58:15.147274 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:15.147242 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:16.015428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.015392 5473 generic.go:356] "Generic (PLEG): container finished" podID="7e500828-6d5d-4b42-816d-41aba06545b7" containerID="a0554e2cfbb3334a85faa9cdb4b2123959d20f82d5435d42923179ba6c535622" exitCode=0 Nov 01 14:58:16.015837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.015481 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" event={"ID":"7e500828-6d5d-4b42-816d-41aba06545b7","Type":"ContainerDied","Data":"a0554e2cfbb3334a85faa9cdb4b2123959d20f82d5435d42923179ba6c535622"} Nov 01 14:58:16.042021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.041987 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt"] Nov 01 14:58:16.042565 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.042552 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" Nov 01 14:58:16.042627 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.042566 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d536e10-e370-47e1-a3ed-122c1b806f5b" containerName="istio-proxy" Nov 01 14:58:16.046025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.046011 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.101650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.101612 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt"] Nov 01 14:58:16.170059 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.170029 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmjrq\" (UniqueName: \"kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.170278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.170252 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.170359 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.170306 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.271330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.271249 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.271330 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.271296 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.271503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.271383 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmjrq\" (UniqueName: \"kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.271955 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.271928 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.273762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.273733 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.331178 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.331153 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmjrq\" (UniqueName: \"kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq\") pod \"maas-api-revoke-sub-fe0c9c71-d47gt\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.359045 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.359021 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:16.372277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.372247 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:16.540531 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.540462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:16.726810 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:16.726784 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71e795f_59fc_4780_b874_3c6611cdd720.slice/crio-0e282c462a05ca961ddf17faa12bf4d29f6aff5a83bdd646279831d756826661 WatchSource:0}: Error finding container 0e282c462a05ca961ddf17faa12bf4d29f6aff5a83bdd646279831d756826661: Status 404 returned error can't find the container with id 0e282c462a05ca961ddf17faa12bf4d29f6aff5a83bdd646279831d756826661 Nov 01 14:58:16.730514 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.730492 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt"] Nov 01 14:58:16.829569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.829497 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28"] Nov 01 14:58:16.829751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.829727 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" containerID="cri-o://ada24c9ecf682ec2fb2e3909ef8560878706457bbc54c0e90f402abdf8e0a1eb" gracePeriod=30 Nov 01 14:58:16.862260 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:16.862207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:17.024727 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.024694 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" event={"ID":"c71e795f-59fc-4780-b874-3c6611cdd720","Type":"ContainerStarted","Data":"0e282c462a05ca961ddf17faa12bf4d29f6aff5a83bdd646279831d756826661"} Nov 01 14:58:17.153497 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.153475 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:17.188035 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.187978 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca\") pod \"7e500828-6d5d-4b42-816d-41aba06545b7\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " Nov 01 14:58:17.188151 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.188124 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca\") pod \"7e500828-6d5d-4b42-816d-41aba06545b7\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " Nov 01 14:58:17.188379 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.188347 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm\" (UniqueName: \"kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm\") pod \"7e500828-6d5d-4b42-816d-41aba06545b7\" (UID: \"7e500828-6d5d-4b42-816d-41aba06545b7\") " Nov 01 14:58:17.188527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.188502 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca" pod "7e500828-6d5d-4b42-816d-41aba06545b7" (UID: "7e500828-6d5d-4b42-816d-41aba06545b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:17.188876 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.188857 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:17.190082 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.190053 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca" pod "7e500828-6d5d-4b42-816d-41aba06545b7" (UID: "7e500828-6d5d-4b42-816d-41aba06545b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:17.190318 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.190296 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm" pod "7e500828-6d5d-4b42-816d-41aba06545b7" (UID: "7e500828-6d5d-4b42-816d-41aba06545b7"). InnerVolumeSpecName "kube-api-access-bl9vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:17.290349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.290319 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl9vm\" (UniqueName: \"kubernetes.io/projected/7e500828-6d5d-4b42-816d-41aba06545b7-kube-api-access-bl9vm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:17.290349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.290347 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e500828-6d5d-4b42-816d-41aba06545b7-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:17.614166 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:17.614131 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:18.033304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:18.033274 5473 generic.go:356] "Generic (PLEG): container finished" podID="c71e795f-59fc-4780-b874-3c6611cdd720" containerID="919e21d34687b99e2199cad6b389cbfcf4ca4e878dfa2b5dc0e4c7dfe2defa8a" exitCode=0 Nov 01 14:58:18.033761 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:18.033309 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" event={"ID":"c71e795f-59fc-4780-b874-3c6611cdd720","Type":"ContainerDied","Data":"919e21d34687b99e2199cad6b389cbfcf4ca4e878dfa2b5dc0e4c7dfe2defa8a"} Nov 01 14:58:18.034815 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:18.034796 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" Nov 01 14:58:18.034933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:18.034816 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67" event={"ID":"7e500828-6d5d-4b42-816d-41aba06545b7","Type":"ContainerDied","Data":"207b68fac784a2a5d6a5d93bd609174c64eb58c037bc950744ac3e4d290dbf63"} Nov 01 14:58:18.034933 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:18.034858 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="207b68fac784a2a5d6a5d93bd609174c64eb58c037bc950744ac3e4d290dbf63" Nov 01 14:58:19.178127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.178099 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:19.212714 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.212686 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca\") pod \"c71e795f-59fc-4780-b874-3c6611cdd720\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " Nov 01 14:58:19.212856 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.212732 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca\") pod \"c71e795f-59fc-4780-b874-3c6611cdd720\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " Nov 01 14:58:19.212929 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.212878 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq\" (UniqueName: \"kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq\") pod \"c71e795f-59fc-4780-b874-3c6611cdd720\" (UID: \"c71e795f-59fc-4780-b874-3c6611cdd720\") " Nov 01 14:58:19.213093 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.213072 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca" pod "c71e795f-59fc-4780-b874-3c6611cdd720" (UID: "c71e795f-59fc-4780-b874-3c6611cdd720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:19.213392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.213374 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:19.214624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.214607 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca" pod "c71e795f-59fc-4780-b874-3c6611cdd720" (UID: "c71e795f-59fc-4780-b874-3c6611cdd720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:19.214851 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.214831 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq" pod "c71e795f-59fc-4780-b874-3c6611cdd720" (UID: "c71e795f-59fc-4780-b874-3c6611cdd720"). InnerVolumeSpecName "kube-api-access-jmjrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:19.314451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.314372 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71e795f-59fc-4780-b874-3c6611cdd720-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:19.314451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:19.314408 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmjrq\" (UniqueName: \"kubernetes.io/projected/c71e795f-59fc-4780-b874-3c6611cdd720-kube-api-access-jmjrq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:20.051285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:20.051246 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" Nov 01 14:58:20.051285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:20.051260 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt" event={"ID":"c71e795f-59fc-4780-b874-3c6611cdd720","Type":"ContainerDied","Data":"0e282c462a05ca961ddf17faa12bf4d29f6aff5a83bdd646279831d756826661"} Nov 01 14:58:20.051496 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:20.051310 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e282c462a05ca961ddf17faa12bf4d29f6aff5a83bdd646279831d756826661" Nov 01 14:58:21.302405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:21.302371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:22.067957 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.067926 5473 generic.go:356] "Generic (PLEG): container finished" podID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerID="ada24c9ecf682ec2fb2e3909ef8560878706457bbc54c0e90f402abdf8e0a1eb" exitCode=0 Nov 01 14:58:22.068071 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.067999 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" event={"ID":"f19b6ee9-20b0-4318-92ab-823cd52cb7dc","Type":"ContainerDied","Data":"ada24c9ecf682ec2fb2e3909ef8560878706457bbc54c0e90f402abdf8e0a1eb"} Nov 01 14:58:22.068122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.068083 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" event={"ID":"f19b6ee9-20b0-4318-92ab-823cd52cb7dc","Type":"ContainerDied","Data":"a3da784f10f1263d05498b165f09cb414681995ce195d84525ad308d7f7b2096"} Nov 01 14:58:22.068122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.068101 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3da784f10f1263d05498b165f09cb414681995ce195d84525ad308d7f7b2096" Nov 01 14:58:22.068122 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.068115 5473 scope.go:122] "RemoveContainer" containerID="8c33ec8c4a63ed0bd11389d709d02dab94294cfd761c14e395dc8a58a504e036" Nov 01 14:58:22.093484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.093463 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:58:22.144988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.144924 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145041 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145111 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145076 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145177 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:22.145264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145180 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145275 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145305 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145330 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145361 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145436 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket\") pod \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\" (UID: \"f19b6ee9-20b0-4318-92ab-823cd52cb7dc\") " Nov 01 14:58:22.145634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145579 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:22.145747 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145662 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:22.145913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145889 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:22.146009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145916 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:22.146009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.145991 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.146139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.146010 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.146139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.146024 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.146139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.146037 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.146139 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.146046 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.147460 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.147433 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:58:22.147770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.147748 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:22.147825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.147757 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "kube-api-access-x6zsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:22.147825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.147772 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token" pod "f19b6ee9-20b0-4318-92ab-823cd52cb7dc" (UID: "f19b6ee9-20b0-4318-92ab-823cd52cb7dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:22.247341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.247316 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6zsv\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-kube-api-access-x6zsv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.247341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.247339 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.247473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.247349 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:22.247473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:22.247358 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19b6ee9-20b0-4318-92ab-823cd52cb7dc-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:23.076869 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:23.076839 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28" Nov 01 14:58:23.309573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:23.309541 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28"] Nov 01 14:58:23.315229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:23.315185 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2048c1-openshift-default-757c847b87-nlf28"] Nov 01 14:58:23.531827 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:23.531796 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:25.082719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:25.082680 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" path="/var/lib/kubelet/pods/f19b6ee9-20b0-4318-92ab-823cd52cb7dc/volumes" Nov 01 14:58:26.244902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:26.244864 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:27.521286 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.521251 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:27.571275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571242 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk"] Nov 01 14:58:27.571824 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571811 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c71e795f-59fc-4780-b874-3c6611cdd720" containerName="revoke-keys" Nov 01 14:58:27.571874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571825 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71e795f-59fc-4780-b874-3c6611cdd720" containerName="revoke-keys" Nov 01 14:58:27.571874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571837 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" Nov 01 14:58:27.571874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571842 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" Nov 01 14:58:27.571874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571861 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7e500828-6d5d-4b42-816d-41aba06545b7" containerName="revoke-keys" Nov 01 14:58:27.571874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571866 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e500828-6d5d-4b42-816d-41aba06545b7" containerName="revoke-keys" Nov 01 14:58:27.572049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571878 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" Nov 01 14:58:27.572049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571882 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" Nov 01 14:58:27.572049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571982 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71e795f-59fc-4780-b874-3c6611cdd720" containerName="revoke-keys" Nov 01 14:58:27.572049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.571994 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" Nov 01 14:58:27.572049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.572002 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e500828-6d5d-4b42-816d-41aba06545b7" containerName="revoke-keys" Nov 01 14:58:27.572049 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.572009 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19b6ee9-20b0-4318-92ab-823cd52cb7dc" containerName="istio-proxy" Nov 01 14:58:27.576953 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.576936 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.630398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.628804 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk"] Nov 01 14:58:27.637758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.637733 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:58:27.674406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.674378 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:58:27.714296 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.714262 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.714451 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.714319 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2np\" (UniqueName: \"kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.714493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.714443 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.815733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.815628 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.816033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.816009 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2np\" (UniqueName: \"kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.816435 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.816409 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.817672 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.817630 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.820870 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.820841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:27.826996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.826966 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.873946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.873917 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2np\" (UniqueName: \"kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np\") pod \"maas-api-revoke-sub-d6e09fec-rffrk\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:27.890195 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:27.890168 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:28.180360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.180279 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm"] Nov 01 14:58:28.201739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.201709 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n"] Nov 01 14:58:28.206947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.206916 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.282117 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:28.279444 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bd3dd2_7523_40bd_98d1_6915aea72761.slice/crio-518e0f035f554004e708c6f4d4560e6456d582c8fb211349b7f24605abaf0277 WatchSource:0}: Error finding container 518e0f035f554004e708c6f4d4560e6456d582c8fb211349b7f24605abaf0277: Status 404 returned error can't find the container with id 518e0f035f554004e708c6f4d4560e6456d582c8fb211349b7f24605abaf0277 Nov 01 14:58:28.282593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.282541 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n"] Nov 01 14:58:28.287116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.287082 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk"] Nov 01 14:58:28.322364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.322342 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.322482 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.322465 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbpzf\" (UniqueName: \"kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.322528 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.322504 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.423859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.423829 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbpzf\" (UniqueName: \"kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.424014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.423877 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.424014 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.423988 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.424671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.424648 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.426375 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.426352 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.447319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.447296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:28.480782 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.480727 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbpzf\" (UniqueName: \"kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf\") pod \"maas-api-revoke-sub-37ffb320-qwx4n\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.521348 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.521322 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:28.621729 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.621697 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:28.903740 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:28.903706 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b17a283_095b_4f13_a69a_75c58c20b806.slice/crio-d7af5ef07a72881b7ffdb007c99f785f455ed3268d8e205f730f57495f6957e4 WatchSource:0}: Error finding container d7af5ef07a72881b7ffdb007c99f785f455ed3268d8e205f730f57495f6957e4: Status 404 returned error can't find the container with id d7af5ef07a72881b7ffdb007c99f785f455ed3268d8e205f730f57495f6957e4 Nov 01 14:58:28.908031 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:28.908003 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n"] Nov 01 14:58:29.120754 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:29.120718 5473 generic.go:356] "Generic (PLEG): container finished" podID="87bd3dd2-7523-40bd-98d1-6915aea72761" containerID="7ad60ff30a4d79a219b2d5679a344383f0f74f4935ec25269fd32c919114470b" exitCode=0 Nov 01 14:58:29.120948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:29.120809 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" event={"ID":"87bd3dd2-7523-40bd-98d1-6915aea72761","Type":"ContainerDied","Data":"7ad60ff30a4d79a219b2d5679a344383f0f74f4935ec25269fd32c919114470b"} Nov 01 14:58:29.120948 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:29.120875 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" event={"ID":"87bd3dd2-7523-40bd-98d1-6915aea72761","Type":"ContainerStarted","Data":"518e0f035f554004e708c6f4d4560e6456d582c8fb211349b7f24605abaf0277"} Nov 01 14:58:29.122271 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:29.122244 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" event={"ID":"5b17a283-095b-4f13-a69a-75c58c20b806","Type":"ContainerStarted","Data":"d7af5ef07a72881b7ffdb007c99f785f455ed3268d8e205f730f57495f6957e4"} Nov 01 14:58:30.132146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.132058 5473 generic.go:356] "Generic (PLEG): container finished" podID="5b17a283-095b-4f13-a69a-75c58c20b806" containerID="63279d001b1581679d32f5108f35ab12e6420b00717a2524f24d25bd1abc5289" exitCode=0 Nov 01 14:58:30.132146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.132132 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" event={"ID":"5b17a283-095b-4f13-a69a-75c58c20b806","Type":"ContainerDied","Data":"63279d001b1581679d32f5108f35ab12e6420b00717a2524f24d25bd1abc5289"} Nov 01 14:58:30.283134 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.283112 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:30.348777 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.348745 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np\" (UniqueName: \"kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np\") pod \"87bd3dd2-7523-40bd-98d1-6915aea72761\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " Nov 01 14:58:30.348978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.348826 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca\") pod \"87bd3dd2-7523-40bd-98d1-6915aea72761\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " Nov 01 14:58:30.348978 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.348928 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca\") pod \"87bd3dd2-7523-40bd-98d1-6915aea72761\" (UID: \"87bd3dd2-7523-40bd-98d1-6915aea72761\") " Nov 01 14:58:30.349186 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.349162 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca" pod "87bd3dd2-7523-40bd-98d1-6915aea72761" (UID: "87bd3dd2-7523-40bd-98d1-6915aea72761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:30.349354 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.349339 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:30.350916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.350895 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np" pod "87bd3dd2-7523-40bd-98d1-6915aea72761" (UID: "87bd3dd2-7523-40bd-98d1-6915aea72761"). InnerVolumeSpecName "kube-api-access-bj2np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:30.350988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.350929 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca" pod "87bd3dd2-7523-40bd-98d1-6915aea72761" (UID: "87bd3dd2-7523-40bd-98d1-6915aea72761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:30.450536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.450457 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj2np\" (UniqueName: \"kubernetes.io/projected/87bd3dd2-7523-40bd-98d1-6915aea72761-kube-api-access-bj2np\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:30.450536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:30.450486 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd3dd2-7523-40bd-98d1-6915aea72761-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:31.108497 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:58:31.108463 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:58:31.140224 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.140178 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 14:58:31.140670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.140233 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" event={"ID":"87bd3dd2-7523-40bd-98d1-6915aea72761","Type":"ContainerDied","Data":"518e0f035f554004e708c6f4d4560e6456d582c8fb211349b7f24605abaf0277"} Nov 01 14:58:31.140670 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.140293 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="518e0f035f554004e708c6f4d4560e6456d582c8fb211349b7f24605abaf0277" Nov 01 14:58:31.280193 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.280166 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:31.360986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.360901 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf\" (UniqueName: \"kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf\") pod \"5b17a283-095b-4f13-a69a-75c58c20b806\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " Nov 01 14:58:31.361140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.361108 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca\") pod \"5b17a283-095b-4f13-a69a-75c58c20b806\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " Nov 01 14:58:31.361239 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.361199 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca\") pod \"5b17a283-095b-4f13-a69a-75c58c20b806\" (UID: \"5b17a283-095b-4f13-a69a-75c58c20b806\") " Nov 01 14:58:31.361527 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.361498 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca" pod "5b17a283-095b-4f13-a69a-75c58c20b806" (UID: "5b17a283-095b-4f13-a69a-75c58c20b806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:31.361730 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.361715 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:31.363226 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.363192 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca" pod "5b17a283-095b-4f13-a69a-75c58c20b806" (UID: "5b17a283-095b-4f13-a69a-75c58c20b806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:31.363329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.363281 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf" pod "5b17a283-095b-4f13-a69a-75c58c20b806" (UID: "5b17a283-095b-4f13-a69a-75c58c20b806"). InnerVolumeSpecName "kube-api-access-tbpzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:31.462964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.462929 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbpzf\" (UniqueName: \"kubernetes.io/projected/5b17a283-095b-4f13-a69a-75c58c20b806-kube-api-access-tbpzf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:31.462964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:31.462957 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b17a283-095b-4f13-a69a-75c58c20b806-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:32.148802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:32.148770 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" Nov 01 14:58:32.149194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:32.148810 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n" event={"ID":"5b17a283-095b-4f13-a69a-75c58c20b806","Type":"ContainerDied","Data":"d7af5ef07a72881b7ffdb007c99f785f455ed3268d8e205f730f57495f6957e4"} Nov 01 14:58:32.149194 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:32.148865 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7af5ef07a72881b7ffdb007c99f785f455ed3268d8e205f730f57495f6957e4" Nov 01 14:58:32.608432 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:32.608398 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:33.163426 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:33.163395 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:33.180874 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:33.180836 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerName="payload-pre-processing" containerID="cri-o://19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d" gracePeriod=25 Nov 01 14:58:36.852310 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:36.852275 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 14:58:37.213522 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.213438 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:37.970969 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.970925 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2"] Nov 01 14:58:37.971756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.971733 5473 cpu_manager.go:414] "containerMap: removing container" podUID="87bd3dd2-7523-40bd-98d1-6915aea72761" containerName="revoke-keys" Nov 01 14:58:37.971872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.971757 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87bd3dd2-7523-40bd-98d1-6915aea72761" containerName="revoke-keys" Nov 01 14:58:37.971872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.971799 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5b17a283-095b-4f13-a69a-75c58c20b806" containerName="revoke-keys" Nov 01 14:58:37.971872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.971808 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b17a283-095b-4f13-a69a-75c58c20b806" containerName="revoke-keys" Nov 01 14:58:37.972050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.971964 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="87bd3dd2-7523-40bd-98d1-6915aea72761" containerName="revoke-keys" Nov 01 14:58:37.972050 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.971991 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b17a283-095b-4f13-a69a-75c58c20b806" containerName="revoke-keys" Nov 01 14:58:37.976288 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:37.976249 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.028859 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.028815 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2"] Nov 01 14:58:38.043710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.043674 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.043710 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.043718 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzmk\" (UniqueName: \"kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.043922 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.043841 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.073655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.073620 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:58:38.073831 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.073704 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:58:38.145074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.145005 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.145343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.145088 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzmk\" (UniqueName: \"kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.145343 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.145180 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.145873 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.145853 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.147586 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.147568 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.208733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.208701 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzmk\" (UniqueName: \"kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk\") pod \"maas-api-revoke-sub-90a5cb9b-49tm2\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.292791 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.292756 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:38.674678 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:38.674637 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec308bb_25e5_4cd8_aea5_7bc1a6be0e24.slice/crio-4298edc25c980ba95a5ec97d3a02612b5ff6a90b3e5cb752570c54497fc4551a WatchSource:0}: Error finding container 4298edc25c980ba95a5ec97d3a02612b5ff6a90b3e5cb752570c54497fc4551a: Status 404 returned error can't find the container with id 4298edc25c980ba95a5ec97d3a02612b5ff6a90b3e5cb752570c54497fc4551a Nov 01 14:58:38.680056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.680033 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2"] Nov 01 14:58:38.948592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.948551 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2"] Nov 01 14:58:38.960234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.955133 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:38.995541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:38.995498 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2"] Nov 01 14:58:39.068518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.068006 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.068518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.068103 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.068518 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.068265 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf7n\" (UniqueName: \"kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.174244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.169996 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.174244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.170116 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.174244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.170358 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf7n\" (UniqueName: \"kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.187242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.182781 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.212289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.211612 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.241248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.239039 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" event={"ID":"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24","Type":"ContainerStarted","Data":"e0398d3822bed6c520238116bd40976cbbbb695018f976fe48e2d4714bcd831d"} Nov 01 14:58:39.241248 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.239122 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" event={"ID":"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24","Type":"ContainerStarted","Data":"4298edc25c980ba95a5ec97d3a02612b5ff6a90b3e5cb752570c54497fc4551a"} Nov 01 14:58:39.277255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.272315 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf7n\" (UniqueName: \"kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n\") pod \"maas-api-revoke-sub-25c6151f-9wrx2\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.376872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.373373 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" podStartSLOduration=2.373350515 podStartE2EDuration="2.373350515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:39.370126863 +0000 UTC m=+2458.428278713" watchObservedRunningTime="2026-11-01 14:58:39.373350515 +0000 UTC m=+2458.431502366" Nov 01 14:58:39.580316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.580258 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:39.857355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:39.856704 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:40.616243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:40.613276 5473 generic.go:356] "Generic (PLEG): container finished" podID="fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" containerID="e0398d3822bed6c520238116bd40976cbbbb695018f976fe48e2d4714bcd831d" exitCode=0 Nov 01 14:58:40.616243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:40.613522 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" event={"ID":"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24","Type":"ContainerDied","Data":"e0398d3822bed6c520238116bd40976cbbbb695018f976fe48e2d4714bcd831d"} Nov 01 14:58:40.723468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:40.723425 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/health\": context deadline exceeded" Nov 01 14:58:41.012241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.010315 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft"] Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.132001 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155020 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lsz\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155159 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155235 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155279 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155312 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155376 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155425 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155457 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.155493 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.211805 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.202579 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2"] Nov 01 14:58:41.224819 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.224734 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft"] Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260031 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260102 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260149 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260239 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260298 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260339 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260380 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260432 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lsz\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.260626 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.262138 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.261295 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.276180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.272413 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.276180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.273078 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.276180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.273336 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.276180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.273563 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.292766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.292716 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.303696 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.302763 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.331289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.331255 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lsz\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.331865 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.331843 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token\") pod \"e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.493740 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.493643 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:41.622703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.622666 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" event={"ID":"1358c224-4d1e-4220-a9b6-5c5f91436d10","Type":"ContainerStarted","Data":"4a503321d71f3886e948e7f04a6c5b8975a6797d8a532b2ba6b7a40a8ff6ae1c"} Nov 01 14:58:41.623148 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.622762 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" event={"ID":"1358c224-4d1e-4220-a9b6-5c5f91436d10","Type":"ContainerStarted","Data":"6deab94fe4722b48755f1692bcd69729f29f5da059ad6c31bb599acd1c972801"} Nov 01 14:58:41.747311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.747267 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" podStartSLOduration=3.747252222 podStartE2EDuration="3.747252222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:41.743194398 +0000 UTC m=+2460.801346248" watchObservedRunningTime="2026-11-01 14:58:41.747252222 +0000 UTC m=+2460.805404072" Nov 01 14:58:41.755273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.755243 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:41.843946 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.843914 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:41.869439 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.869403 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk\" (UniqueName: \"kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk\") pod \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " Nov 01 14:58:41.869641 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.869559 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca\") pod \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " Nov 01 14:58:41.869715 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.869653 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca\") pod \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\" (UID: \"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24\") " Nov 01 14:58:41.870709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.870619 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca" pod "fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" (UID: "fec308bb-25e5-4cd8-aea5-7bc1a6be0e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:41.873027 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.872996 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca" pod "fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" (UID: "fec308bb-25e5-4cd8-aea5-7bc1a6be0e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:41.873198 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.873182 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk" pod "fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" (UID: "fec308bb-25e5-4cd8-aea5-7bc1a6be0e24"). InnerVolumeSpecName "kube-api-access-qfzmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:41.917801 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:41.917775 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26b9161_8159_469b_ac44_718a9632746d.slice/crio-6949ec8ca32fb79da2371c3d036735d914b8abe3b2f810f7086439055e577bcf WatchSource:0}: Error finding container 6949ec8ca32fb79da2371c3d036735d914b8abe3b2f810f7086439055e577bcf: Status 404 returned error can't find the container with id 6949ec8ca32fb79da2371c3d036735d914b8abe3b2f810f7086439055e577bcf Nov 01 14:58:41.920398 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.920368 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:58:41.920484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.920434 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:58:41.920484 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.920466 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:58:41.922209 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.922185 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft"] Nov 01 14:58:41.971651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.971626 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:41.971651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.971651 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzmk\" (UniqueName: \"kubernetes.io/projected/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-kube-api-access-qfzmk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:41.971813 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:41.971663 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:42.631799 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.631767 5473 generic.go:356] "Generic (PLEG): container finished" podID="1358c224-4d1e-4220-a9b6-5c5f91436d10" containerID="4a503321d71f3886e948e7f04a6c5b8975a6797d8a532b2ba6b7a40a8ff6ae1c" exitCode=0 Nov 01 14:58:42.632316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.631853 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" event={"ID":"1358c224-4d1e-4220-a9b6-5c5f91436d10","Type":"ContainerDied","Data":"4a503321d71f3886e948e7f04a6c5b8975a6797d8a532b2ba6b7a40a8ff6ae1c"} Nov 01 14:58:42.633634 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.633611 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" event={"ID":"f26b9161-8159-469b-ac44-718a9632746d","Type":"ContainerStarted","Data":"da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674"} Nov 01 14:58:42.633773 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.633655 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" event={"ID":"f26b9161-8159-469b-ac44-718a9632746d","Type":"ContainerStarted","Data":"6949ec8ca32fb79da2371c3d036735d914b8abe3b2f810f7086439055e577bcf"} Nov 01 14:58:42.634958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.634940 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" Nov 01 14:58:42.634958 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.634960 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2" event={"ID":"fec308bb-25e5-4cd8-aea5-7bc1a6be0e24","Type":"ContainerDied","Data":"4298edc25c980ba95a5ec97d3a02612b5ff6a90b3e5cb752570c54497fc4551a"} Nov 01 14:58:42.635126 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.634997 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4298edc25c980ba95a5ec97d3a02612b5ff6a90b3e5cb752570c54497fc4551a" Nov 01 14:58:42.865818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.865766 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podStartSLOduration=2.86575138 podStartE2EDuration="2.86575138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:42.862720075 +0000 UTC m=+2461.920871925" watchObservedRunningTime="2026-11-01 14:58:42.86575138 +0000 UTC m=+2461.923903244" Nov 01 14:58:42.871163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:42.871132 5473 scope.go:122] "RemoveContainer" containerID="ada24c9ecf682ec2fb2e3909ef8560878706457bbc54c0e90f402abdf8e0a1eb" Nov 01 14:58:43.494264 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.494238 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:43.495956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.495869 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 14:58:43.496033 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.495995 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 14:58:43.860861 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.860836 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:43.907337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.907302 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca\") pod \"1358c224-4d1e-4220-a9b6-5c5f91436d10\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " Nov 01 14:58:43.907503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.907347 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca\") pod \"1358c224-4d1e-4220-a9b6-5c5f91436d10\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " Nov 01 14:58:43.907503 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.907390 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n\" (UniqueName: \"kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n\") pod \"1358c224-4d1e-4220-a9b6-5c5f91436d10\" (UID: \"1358c224-4d1e-4220-a9b6-5c5f91436d10\") " Nov 01 14:58:43.907651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.907629 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca" pod "1358c224-4d1e-4220-a9b6-5c5f91436d10" (UID: "1358c224-4d1e-4220-a9b6-5c5f91436d10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:43.907960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.907931 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:43.909517 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.909489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n" pod "1358c224-4d1e-4220-a9b6-5c5f91436d10" (UID: "1358c224-4d1e-4220-a9b6-5c5f91436d10"). InnerVolumeSpecName "kube-api-access-qcf7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:43.909624 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:43.909587 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca" pod "1358c224-4d1e-4220-a9b6-5c5f91436d10" (UID: "1358c224-4d1e-4220-a9b6-5c5f91436d10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:44.009388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.009348 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1358c224-4d1e-4220-a9b6-5c5f91436d10-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:44.009388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.009383 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcf7n\" (UniqueName: \"kubernetes.io/projected/1358c224-4d1e-4220-a9b6-5c5f91436d10-kube-api-access-qcf7n\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:44.494366 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.494332 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 14:58:44.494542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.494386 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 14:58:44.664360 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.664328 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" Nov 01 14:58:44.664544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.664355 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2" event={"ID":"1358c224-4d1e-4220-a9b6-5c5f91436d10","Type":"ContainerDied","Data":"6deab94fe4722b48755f1692bcd69729f29f5da059ad6c31bb599acd1c972801"} Nov 01 14:58:44.664544 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:44.664413 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6deab94fe4722b48755f1692bcd69729f29f5da059ad6c31bb599acd1c972801" Nov 01 14:58:45.494275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:45.494211 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 14:58:45.494757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:45.494304 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 14:58:45.671020 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:45.670985 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:46.495335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:46.494547 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 14:58:46.495335 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:46.494606 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 14:58:46.750492 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:46.750406 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:46.875000 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:46.874960 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 14:58:47.494597 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:47.494557 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 14:58:47.494830 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:47.494621 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 14:58:48.377328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:48.377296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:48.495084 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:48.495044 5473 patch_prober.go:29] interesting pod/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 01 14:58:48.495284 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:48.495127 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 01 14:58:48.538402 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:48.538351 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:49.520406 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:49.520370 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:49.520890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:49.520751 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:49.522118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:49.522100 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 14:58:50.320147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:50.320117 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:50.848242 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:50.848181 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:51.282361 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:51.282291 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:52.898364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:52.898330 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:53.434524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.434501 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:58:53.478190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.478101 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume\") pod \"41870ba8-a6fb-405e-b6eb-4a80222a785f\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " Nov 01 14:58:53.478190 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.478153 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8\" (UniqueName: \"kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8\") pod \"41870ba8-a6fb-405e-b6eb-4a80222a785f\" (UID: \"41870ba8-a6fb-405e-b6eb-4a80222a785f\") " Nov 01 14:58:53.478552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.478523 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume" pod "41870ba8-a6fb-405e-b6eb-4a80222a785f" (UID: "41870ba8-a6fb-405e-b6eb-4a80222a785f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:53.478796 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.478770 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41870ba8-a6fb-405e-b6eb-4a80222a785f-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:53.480594 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.480553 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8" pod "41870ba8-a6fb-405e-b6eb-4a80222a785f" (UID: "41870ba8-a6fb-405e-b6eb-4a80222a785f"). InnerVolumeSpecName "kube-api-access-fqrq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:53.485029 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.485003 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7"] Nov 01 14:58:53.485298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.485271 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" containerID="cri-o://3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0" gracePeriod=30 Nov 01 14:58:53.579901 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.579869 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqrq8\" (UniqueName: \"kubernetes.io/projected/41870ba8-a6fb-405e-b6eb-4a80222a785f-kube-api-access-fqrq8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:53.737357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.737273 5473 generic.go:356] "Generic (PLEG): container finished" podID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerID="19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d" exitCode=0 Nov 01 14:58:53.737357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.737317 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" Nov 01 14:58:53.737540 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.737359 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" event={"ID":"41870ba8-a6fb-405e-b6eb-4a80222a785f","Type":"ContainerDied","Data":"19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d"} Nov 01 14:58:53.737540 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.737414 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm" event={"ID":"41870ba8-a6fb-405e-b6eb-4a80222a785f","Type":"ContainerDied","Data":"fce3e845f0e5e6e0e29045a9096ca304e6ed14d0daf77d4710875c254e5b4817"} Nov 01 14:58:53.737540 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.737440 5473 scope.go:122] "RemoveContainer" containerID="19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d" Nov 01 14:58:53.748523 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.748502 5473 scope.go:122] "RemoveContainer" containerID="19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d" Nov 01 14:58:53.752370 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:58:53.751321 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d\": container with ID starting with 19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d not found: ID does not exist" containerID="19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d" Nov 01 14:58:53.752370 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.751358 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d"} err="failed to get container status \"19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d\": rpc error: code = NotFound desc = could not find container \"19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d\": container with ID starting with 19d2e9027148cf2185591ae721c64712fb09f74b85575a07273070165a76731d not found: ID does not exist" Nov 01 14:58:53.913153 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.913108 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm"] Nov 01 14:58:53.916883 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:53.916853 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f764c7-66b585d4dc-7dcjm"] Nov 01 14:58:54.116170 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:54.116140 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:54.291621 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:54.291588 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:55.082389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.082360 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" path="/var/lib/kubelet/pods/41870ba8-a6fb-405e-b6eb-4a80222a785f/volumes" Nov 01 14:58:55.226387 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.226343 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:55.394295 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.394170 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:55.558617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.558564 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:55.692677 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.692608 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr"] Nov 01 14:58:55.693162 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693148 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1358c224-4d1e-4220-a9b6-5c5f91436d10" containerName="revoke-keys" Nov 01 14:58:55.693208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693164 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1358c224-4d1e-4220-a9b6-5c5f91436d10" containerName="revoke-keys" Nov 01 14:58:55.693208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693176 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" containerName="revoke-keys" Nov 01 14:58:55.693208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693181 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" containerName="revoke-keys" Nov 01 14:58:55.693208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693198 5473 cpu_manager.go:414] "containerMap: removing container" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerName="payload-pre-processing" Nov 01 14:58:55.693208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693203 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerName="payload-pre-processing" Nov 01 14:58:55.693409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693304 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="41870ba8-a6fb-405e-b6eb-4a80222a785f" containerName="payload-pre-processing" Nov 01 14:58:55.693409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693319 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1358c224-4d1e-4220-a9b6-5c5f91436d10" containerName="revoke-keys" Nov 01 14:58:55.693409 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.693325 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" containerName="revoke-keys" Nov 01 14:58:55.698400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.698382 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.749916 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.749883 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr"] Nov 01 14:58:55.794319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.794291 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:58:55.794490 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.794357 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:58:55.808853 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.808829 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.809038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.808859 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.809038 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.808886 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2kdw\" (UniqueName: \"kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.907129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.907090 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b"] Nov 01 14:58:55.907357 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.907329 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" containerID="cri-o://a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4" gracePeriod=30 Nov 01 14:58:55.911147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.911117 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.911467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.911163 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.911467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.911202 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2kdw\" (UniqueName: \"kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.911921 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.911899 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.913919 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.913893 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.971298 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.971237 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2kdw\" (UniqueName: \"kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw\") pod \"maas-api-revoke-sub-1e421592-9qzkr\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:55.977795 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:55.977772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:56.012069 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.012039 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:56.144990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.144963 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:56.435818 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:56.435789 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca7b9ae_a55a_4d7f_b73c_958824fce64b.slice/crio-48902e173dc7b807c390db0ae5e9f407e6e794fa8684f0ba121aa31829b9866e WatchSource:0}: Error finding container 48902e173dc7b807c390db0ae5e9f407e6e794fa8684f0ba121aa31829b9866e: Status 404 returned error can't find the container with id 48902e173dc7b807c390db0ae5e9f407e6e794fa8684f0ba121aa31829b9866e Nov 01 14:58:56.440828 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.440809 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr"] Nov 01 14:58:56.698703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.698632 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:56.763860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.763831 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" event={"ID":"8ca7b9ae-a55a-4d7f-b73c-958824fce64b","Type":"ContainerStarted","Data":"5c3134a03f7561fa60895d7e2846efbea21dac08ea4407968e8231e7631b7579"} Nov 01 14:58:56.763999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.763882 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" event={"ID":"8ca7b9ae-a55a-4d7f-b73c-958824fce64b","Type":"ContainerStarted","Data":"48902e173dc7b807c390db0ae5e9f407e6e794fa8684f0ba121aa31829b9866e"} Nov 01 14:58:56.878337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.878290 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:56.882915 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:56.882871 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" podStartSLOduration=1.882857913 podStartE2EDuration="1.882857913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:56.881419554 +0000 UTC m=+2475.939571405" watchObservedRunningTime="2026-11-01 14:58:56.882857913 +0000 UTC m=+2475.941009763" Nov 01 14:58:57.318848 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.318819 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm"] Nov 01 14:58:57.319180 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.319023 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" containerID="cri-o://657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b" gracePeriod=30 Nov 01 14:58:57.346724 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.346697 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:57.520809 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.520776 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:57.693834 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.693753 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:57.772905 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.772870 5473 generic.go:356] "Generic (PLEG): container finished" podID="8ca7b9ae-a55a-4d7f-b73c-958824fce64b" containerID="5c3134a03f7561fa60895d7e2846efbea21dac08ea4407968e8231e7631b7579" exitCode=0 Nov 01 14:58:57.773072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.772966 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" event={"ID":"8ca7b9ae-a55a-4d7f-b73c-958824fce64b","Type":"ContainerDied","Data":"5c3134a03f7561fa60895d7e2846efbea21dac08ea4407968e8231e7631b7579"} Nov 01 14:58:57.861146 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:57.861109 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:58.314552 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.314519 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:58.758590 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.757886 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:58:58.781316 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.781285 5473 generic.go:356] "Generic (PLEG): container finished" podID="61ed7157-72cf-49b8-b90c-a01470139213" containerID="3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0" exitCode=0 Nov 01 14:58:58.781470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.781332 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" Nov 01 14:58:58.781470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.781371 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" event={"ID":"61ed7157-72cf-49b8-b90c-a01470139213","Type":"ContainerDied","Data":"3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0"} Nov 01 14:58:58.781470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.781451 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7" event={"ID":"61ed7157-72cf-49b8-b90c-a01470139213","Type":"ContainerDied","Data":"f0e80f4b4675db9805a9d4592b029a044cd882a164ab2a724b906baca8551cba"} Nov 01 14:58:58.781599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.781475 5473 scope.go:122] "RemoveContainer" containerID="3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0" Nov 01 14:58:58.792097 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.792078 5473 scope.go:122] "RemoveContainer" containerID="e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08" Nov 01 14:58:58.793118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.793056 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl"] Nov 01 14:58:58.793935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.793915 5473 cpu_manager.go:414] "containerMap: removing container" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" Nov 01 14:58:58.794009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.793939 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" Nov 01 14:58:58.794116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.794104 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" Nov 01 14:58:58.799652 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.799634 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.803068 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.802990 5473 scope.go:122] "RemoveContainer" containerID="3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0" Nov 01 14:58:58.803305 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:58:58.803285 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0\": container with ID starting with 3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0 not found: ID does not exist" containerID="3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0" Nov 01 14:58:58.803407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.803316 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0"} err="failed to get container status \"3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0\": rpc error: code = NotFound desc = could not find container \"3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0\": container with ID starting with 3293699cb2513e218b350be6956702c17396328e825bc9c734c2b27939c4d3e0 not found: ID does not exist" Nov 01 14:58:58.803407 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.803353 5473 scope.go:122] "RemoveContainer" containerID="e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08" Nov 01 14:58:58.803655 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:58:58.803628 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08\": container with ID starting with e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08 not found: ID does not exist" containerID="e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08" Nov 01 14:58:58.803787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.803661 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08"} err="failed to get container status \"e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08\": rpc error: code = NotFound desc = could not find container \"e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08\": container with ID starting with e4bc7894803b94c83319b18c280dd669fa2c6ac7e8181c5a7a6c00889f7d1a08 not found: ID does not exist" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864180 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864240 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864318 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864393 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864454 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864518 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864539 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864560 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864622 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo\") pod \"61ed7157-72cf-49b8-b90c-a01470139213\" (UID: \"61ed7157-72cf-49b8-b90c-a01470139213\") " Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864812 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864854 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864871 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864909 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz98d\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864933 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.864963 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.865026 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.865090 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.865127 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.871048 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.871257 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.872561 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.873115 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:58.874347 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.873856 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:58.875823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.875783 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:58:58.880389 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.880308 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:58:58.882132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.882095 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:58.884304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.884281 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2" pod "61ed7157-72cf-49b8-b90c-a01470139213" (UID: "61ed7157-72cf-49b8-b90c-a01470139213"). InnerVolumeSpecName "kube-api-access-4rnh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:58.893256 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.893187 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-97c87c-openshift-default-dockercfg-6msg6\"" Nov 01 14:58:58.909430 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.909408 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl"] Nov 01 14:58:58.912241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.912180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:58.966581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966542 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz98d\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.966763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966615 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.966763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966665 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.966763 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966744 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.966911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966818 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.966911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966864 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.966988 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.966969 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.967032 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967019 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.967074 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967044 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.967127 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967107 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ed7157-72cf-49b8-b90c-a01470139213-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967128 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rnh2\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-kube-api-access-4rnh2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967145 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967160 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ed7157-72cf-49b8-b90c-a01470139213-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967176 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967190 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ed7157-72cf-49b8-b90c-a01470139213-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967203 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967250 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967267 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967266 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ed7157-72cf-49b8-b90c-a01470139213-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:58.967720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967493 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.967838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967784 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.967838 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.967810 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.968243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.968188 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.968673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.968650 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.969812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.969783 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.969812 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.969798 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:58.990234 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:58.990194 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:59.028042 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.028013 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz98d\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:59.028758 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.028735 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token\") pod \"e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:59.068079 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.068046 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw\" (UniqueName: \"kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw\") pod \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " Nov 01 14:58:59.068289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.068091 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca\") pod \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " Nov 01 14:58:59.068388 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.068325 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca\") pod \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\" (UID: \"8ca7b9ae-a55a-4d7f-b73c-958824fce64b\") " Nov 01 14:58:59.068592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.068560 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca" pod "8ca7b9ae-a55a-4d7f-b73c-958824fce64b" (UID: "8ca7b9ae-a55a-4d7f-b73c-958824fce64b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:58:59.069085 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.069056 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:59.071857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.071829 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw" pod "8ca7b9ae-a55a-4d7f-b73c-958824fce64b" (UID: "8ca7b9ae-a55a-4d7f-b73c-958824fce64b"). InnerVolumeSpecName "kube-api-access-c2kdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:58:59.071964 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.071868 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca" pod "8ca7b9ae-a55a-4d7f-b73c-958824fce64b" (UID: "8ca7b9ae-a55a-4d7f-b73c-958824fce64b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:58:59.093404 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.093368 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:59.156825 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.156509 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:58:59.170909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.170876 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:59.170909 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.170908 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2kdw\" (UniqueName: \"kubernetes.io/projected/8ca7b9ae-a55a-4d7f-b73c-958824fce64b-kube-api-access-c2kdw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:58:59.283073 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.283036 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:58:59.316462 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.316432 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7"] Nov 01 14:58:59.322205 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.322182 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-288f1e-openshift-default-7588785f49-rhfr7"] Nov 01 14:58:59.594102 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:58:59.594068 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093f8eb7_53cb_4245_9a53_888b2f8c8b4c.slice/crio-95674e1f48cf3b448ac186af8ab1351972c930990fc6c5e028a442779803f58e WatchSource:0}: Error finding container 95674e1f48cf3b448ac186af8ab1351972c930990fc6c5e028a442779803f58e: Status 404 returned error can't find the container with id 95674e1f48cf3b448ac186af8ab1351972c930990fc6c5e028a442779803f58e Nov 01 14:58:59.597208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.597167 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:58:59.597368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.597269 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:58:59.597368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.597336 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:58:59.598443 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.598426 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl"] Nov 01 14:58:59.792185 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.792150 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" event={"ID":"8ca7b9ae-a55a-4d7f-b73c-958824fce64b","Type":"ContainerDied","Data":"48902e173dc7b807c390db0ae5e9f407e6e794fa8684f0ba121aa31829b9866e"} Nov 01 14:58:59.792650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.792204 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48902e173dc7b807c390db0ae5e9f407e6e794fa8684f0ba121aa31829b9866e" Nov 01 14:58:59.792650 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.792158 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr" Nov 01 14:58:59.794184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.794155 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" event={"ID":"093f8eb7-53cb-4245-9a53-888b2f8c8b4c","Type":"ContainerStarted","Data":"7199649c39a12ee9165c54518af6622d80b959797167204c761c903ec54df7a4"} Nov 01 14:58:59.794417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.794205 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" event={"ID":"093f8eb7-53cb-4245-9a53-888b2f8c8b4c","Type":"ContainerStarted","Data":"95674e1f48cf3b448ac186af8ab1351972c930990fc6c5e028a442779803f58e"} Nov 01 14:58:59.910614 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:58:59.910562 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podStartSLOduration=1.910548878 podStartE2EDuration="1.910548878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:58:59.91012709 +0000 UTC m=+2478.968278941" watchObservedRunningTime="2026-11-01 14:58:59.910548878 +0000 UTC m=+2478.968700728" Nov 01 14:59:00.157278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.157184 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:59:00.164822 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.164795 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:59:00.585392 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.585357 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:00.745376 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.745335 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:00.802005 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.801976 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:59:00.802894 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.802875 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 14:59:00.964902 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.964873 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2"] Nov 01 14:59:00.965446 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.965432 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8ca7b9ae-a55a-4d7f-b73c-958824fce64b" containerName="revoke-keys" Nov 01 14:59:00.965509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.965449 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca7b9ae-a55a-4d7f-b73c-958824fce64b" containerName="revoke-keys" Nov 01 14:59:00.965509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.965469 5473 cpu_manager.go:414] "containerMap: removing container" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" Nov 01 14:59:00.965509 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.965474 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" Nov 01 14:59:00.965608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.965561 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ed7157-72cf-49b8-b90c-a01470139213" containerName="istio-proxy" Nov 01 14:59:00.965608 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.965573 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca7b9ae-a55a-4d7f-b73c-958824fce64b" containerName="revoke-keys" Nov 01 14:59:00.969025 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:00.969003 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.021294 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.021267 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4e07cb-dockercfg-47fpz\"" Nov 01 14:59:01.021452 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.021293 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4e07cb\"" Nov 01 14:59:01.021536 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.021500 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2"] Nov 01 14:59:01.149414 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.149381 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ed7157-72cf-49b8-b90c-a01470139213" path="/var/lib/kubelet/pods/61ed7157-72cf-49b8-b90c-a01470139213/volumes" Nov 01 14:59:01.150837 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.150803 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq"] Nov 01 14:59:01.164500 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.164474 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.165513 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:01.165061 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:59:01.185618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.185583 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq"] Nov 01 14:59:01.224055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.224021 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.224307 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.224285 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hs8\" (UniqueName: \"kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8\") pod \"payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.257546 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.257524 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:59:01.325285 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.325203 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hs8\" (UniqueName: \"kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8\") pod \"payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.325468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.325338 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.325468 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.325389 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.325569 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.325537 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hqk\" (UniqueName: \"kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk\") pod \"payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.326258 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.326236 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.383596 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.383556 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hs8\" (UniqueName: \"kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8\") pod \"payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.426417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426345 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426417 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426392 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426450 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426482 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426506 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426605 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426566 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426598 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:01.426817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426655 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426701 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426742 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert\") pod \"300f8c59-371c-4990-8d63-48069e9f87fb\" (UID: \"300f8c59-371c-4990-8d63-48069e9f87fb\") " Nov 01 14:59:01.426817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426807 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:01.427088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426839 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:01.427088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426913 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:01.427088 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.426978 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.427273 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427235 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:59:01.427332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427313 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hqk\" (UniqueName: \"kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk\") pod \"payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.427513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427493 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.427618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427519 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.427618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427536 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.427618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427551 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.427618 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427565 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300f8c59-371c-4990-8d63-48069e9f87fb-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.427872 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.427848 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.429009 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.428986 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:01.429120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.429010 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:01.429120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.429030 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:59:01.429199 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.429148 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44" pod "300f8c59-371c-4990-8d63-48069e9f87fb" (UID: "300f8c59-371c-4990-8d63-48069e9f87fb"). InnerVolumeSpecName "kube-api-access-njv44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:01.484413 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.484383 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hqk\" (UniqueName: \"kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk\") pod \"payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.488184 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.488160 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:01.528405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.528373 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300f8c59-371c-4990-8d63-48069e9f87fb-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.528405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.528401 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300f8c59-371c-4990-8d63-48069e9f87fb-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.528405 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.528411 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njv44\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-kube-api-access-njv44\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.528633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.528420 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300f8c59-371c-4990-8d63-48069e9f87fb-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:01.594076 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.594045 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:01.811733 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.811704 5473 generic.go:356] "Generic (PLEG): container finished" podID="300f8c59-371c-4990-8d63-48069e9f87fb" containerID="a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4" exitCode=0 Nov 01 14:59:01.812128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.811759 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" Nov 01 14:59:01.812128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.811786 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" event={"ID":"300f8c59-371c-4990-8d63-48069e9f87fb","Type":"ContainerDied","Data":"a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4"} Nov 01 14:59:01.812128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.811838 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b" event={"ID":"300f8c59-371c-4990-8d63-48069e9f87fb","Type":"ContainerDied","Data":"c23fc4f64a2f5c09e9374387549f3bb93183277238f19fc207754f901b99da90"} Nov 01 14:59:01.812128 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.811868 5473 scope.go:122] "RemoveContainer" containerID="a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4" Nov 01 14:59:01.824766 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.824749 5473 scope.go:122] "RemoveContainer" containerID="bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864" Nov 01 14:59:01.834424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.834398 5473 scope.go:122] "RemoveContainer" containerID="a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4" Nov 01 14:59:01.834673 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:01.834654 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4\": container with ID starting with a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4 not found: ID does not exist" containerID="a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4" Nov 01 14:59:01.834762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.834682 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4"} err="failed to get container status \"a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4\": rpc error: code = NotFound desc = could not find container \"a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4\": container with ID starting with a53c6727e6a849f2701cf870df5ea59c8be84fa4337926dcbfcf1f6ea13f10d4 not found: ID does not exist" Nov 01 14:59:01.834762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.834699 5473 scope.go:122] "RemoveContainer" containerID="bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864" Nov 01 14:59:01.834984 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:01.834964 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864\": container with ID starting with bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864 not found: ID does not exist" containerID="bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864" Nov 01 14:59:01.835030 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.834991 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864"} err="failed to get container status \"bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864\": rpc error: code = NotFound desc = could not find container \"bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864\": container with ID starting with bb3947821c4e1d82ba9b506b04b673b8dbda82f187153609b5b482a57fbf7864 not found: ID does not exist" Nov 01 14:59:01.861114 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:59:01.861089 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode598863a_d16a_4959_9a78_3e73ed28ee5a.slice/crio-24209f5799f6f0042030f624870cc8c1e1560767f6a10a9caf7e656c88b54d2d WatchSource:0}: Error finding container 24209f5799f6f0042030f624870cc8c1e1560767f6a10a9caf7e656c88b54d2d: Status 404 returned error can't find the container with id 24209f5799f6f0042030f624870cc8c1e1560767f6a10a9caf7e656c88b54d2d Nov 01 14:59:01.865463 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.865439 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq"] Nov 01 14:59:01.966428 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:59:01.966401 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6047ed06_dc73_49f1_8897_129f144c0afb.slice/crio-7024441c6ee57e33ec1650d174648d9ba3e715c888a302dd710fff9cad57a975 WatchSource:0}: Error finding container 7024441c6ee57e33ec1650d174648d9ba3e715c888a302dd710fff9cad57a975: Status 404 returned error can't find the container with id 7024441c6ee57e33ec1650d174648d9ba3e715c888a302dd710fff9cad57a975 Nov 01 14:59:01.970947 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.970921 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2"] Nov 01 14:59:01.983818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:01.983790 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b"] Nov 01 14:59:02.027208 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.027179 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f764c7-openshift-default-f554779df-9fr9b"] Nov 01 14:59:02.218444 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.218367 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:02.606890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.606861 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:59:02.690573 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.690543 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:02.745699 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745659 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.745871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745709 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.745871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745782 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.745871 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745834 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.746081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745885 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.746081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745918 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.746081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.745980 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.746081 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.746063 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.746341 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.746174 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket\") pod \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\" (UID: \"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32\") " Nov 01 14:59:02.748823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.747294 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:02.748823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.747483 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:02.748823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.747716 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:02.748823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.748127 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:59:02.748823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.748420 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:02.749408 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.749343 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:02.749823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.749802 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "kube-api-access-29lrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:02.750177 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.750160 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:02.750305 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.750271 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo" pod "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" (UID: "3cc9c369-f74e-4f5a-b5e6-ae238c5cba32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:59:02.824116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.824024 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" event={"ID":"e598863a-d16a-4959-9a78-3e73ed28ee5a","Type":"ContainerStarted","Data":"ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4"} Nov 01 14:59:02.824116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.824086 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:02.824116 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.824112 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" event={"ID":"e598863a-d16a-4959-9a78-3e73ed28ee5a","Type":"ContainerStarted","Data":"24209f5799f6f0042030f624870cc8c1e1560767f6a10a9caf7e656c88b54d2d"} Nov 01 14:59:02.825737 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.825711 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" event={"ID":"6047ed06-dc73-49f1-8897-129f144c0afb","Type":"ContainerStarted","Data":"3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f"} Nov 01 14:59:02.825884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.825755 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" event={"ID":"6047ed06-dc73-49f1-8897-129f144c0afb","Type":"ContainerStarted","Data":"7024441c6ee57e33ec1650d174648d9ba3e715c888a302dd710fff9cad57a975"} Nov 01 14:59:02.825884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.825782 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:02.827393 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.827366 5473 generic.go:356] "Generic (PLEG): container finished" podID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerID="657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b" exitCode=0 Nov 01 14:59:02.827513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.827402 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" Nov 01 14:59:02.827513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.827414 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" event={"ID":"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32","Type":"ContainerDied","Data":"657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b"} Nov 01 14:59:02.827513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.827454 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm" event={"ID":"3cc9c369-f74e-4f5a-b5e6-ae238c5cba32","Type":"ContainerDied","Data":"506f85f9fbc158fca00bf24ebdd9f5f74e63bce522621ad57c47012b1cdea724"} Nov 01 14:59:02.827513 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.827500 5473 scope.go:122] "RemoveContainer" containerID="657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b" Nov 01 14:59:02.841289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.841268 5473 scope.go:122] "RemoveContainer" containerID="ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d" Nov 01 14:59:02.847846 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847825 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29lrm\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-kube-api-access-29lrm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847849 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847860 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847868 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847878 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847886 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847895 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847904 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.847931 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.847912 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:02.851131 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.851116 5473 scope.go:122] "RemoveContainer" containerID="657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b" Nov 01 14:59:02.851416 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:02.851397 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b\": container with ID starting with 657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b not found: ID does not exist" containerID="657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b" Nov 01 14:59:02.851493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.851424 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b"} err="failed to get container status \"657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b\": rpc error: code = NotFound desc = could not find container \"657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b\": container with ID starting with 657f29ba08de721193b8aa01164e9f805e0d005b9a535412ee124a0739adba9b not found: ID does not exist" Nov 01 14:59:02.851493 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.851440 5473 scope.go:122] "RemoveContainer" containerID="ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d" Nov 01 14:59:02.851698 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:02.851681 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d\": container with ID starting with ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d not found: ID does not exist" containerID="ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d" Nov 01 14:59:02.851743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.851701 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d"} err="failed to get container status \"ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d\": rpc error: code = NotFound desc = could not find container \"ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d\": container with ID starting with ca685404175ce0ef12b3d8dbc54b0fc01516e6183a51c56defe0c2487e68186d not found: ID does not exist" Nov 01 14:59:02.870249 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.870209 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:02.941892 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:02.941846 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" podStartSLOduration=1.9418325269999999 podStartE2EDuration="1.941832527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:02.937321093 +0000 UTC m=+2481.995472943" watchObservedRunningTime="2026-11-01 14:59:02.941832527 +0000 UTC m=+2481.999984376" Nov 01 14:59:03.041695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.041656 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:03.052333 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.052305 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm"] Nov 01 14:59:03.056136 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.056116 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-7197e2-openshift-default-c7f95f7b9-zlwqm"] Nov 01 14:59:03.083104 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.083031 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" path="/var/lib/kubelet/pods/300f8c59-371c-4990-8d63-48069e9f87fb/volumes" Nov 01 14:59:03.083651 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.083630 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" path="/var/lib/kubelet/pods/3cc9c369-f74e-4f5a-b5e6-ae238c5cba32/volumes" Nov 01 14:59:03.177141 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.177087 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" podStartSLOduration=3.177072819 podStartE2EDuration="3.177072819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:03.174207045 +0000 UTC m=+2482.232358897" watchObservedRunningTime="2026-11-01 14:59:03.177072819 +0000 UTC m=+2482.235224669" Nov 01 14:59:03.758785 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.758749 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:03.823599 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.823564 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj"] Nov 01 14:59:03.922140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:03.922106 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t"] Nov 01 14:59:04.039997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:04.039956 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:04.204742 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:04.204705 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:04.403843 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:04.403764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:05.237882 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:05.237851 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:05.421291 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:05.421253 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:06.807437 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:06.807399 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:07.094277 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:07.094168 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:07.294709 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:07.294666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:07.481935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:07.481840 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:07.701644 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:07.701609 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:08.011241 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:07.975310 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l"] Nov 01 14:59:08.048261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.013057 5473 cpu_manager.go:414] "containerMap: removing container" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" Nov 01 14:59:08.048261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.013089 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" Nov 01 14:59:08.048261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.013131 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" Nov 01 14:59:08.048261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.013140 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" Nov 01 14:59:08.048261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.013151 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" Nov 01 14:59:08.075039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.069474 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" Nov 01 14:59:08.075039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.069578 5473 cpu_manager.go:414] "containerMap: removing container" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" Nov 01 14:59:08.075039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.069591 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" Nov 01 14:59:08.189325 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.108282 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" Nov 01 14:59:08.237372 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.237335 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" Nov 01 14:59:08.245245 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.237601 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc9c369-f74e-4f5a-b5e6-ae238c5cba32" containerName="istio-proxy" Nov 01 14:59:08.321327 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.321229 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:08.497247 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.490587 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.498374 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.498348 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l"] Nov 01 14:59:08.506332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.506306 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t"] Nov 01 14:59:08.535938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.535905 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="300f8c59-371c-4990-8d63-48069e9f87fb" containerName="istio-proxy" Nov 01 14:59:08.540424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.540399 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t"] Nov 01 14:59:08.540720 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.540704 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.543734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.543713 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-97c87c\"" Nov 01 14:59:08.543924 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.543726 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-97c87c-dockercfg-bbd2m\"" Nov 01 14:59:08.594278 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.588931 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.594587 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.594564 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.594927 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.594906 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.595743 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.595722 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.595917 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.595899 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.596080 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.596063 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.612384 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.596893 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.623147 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.623120 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.623457 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.623436 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.623663 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.623645 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psljl\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.623841 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.623824 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p74p5\" (UniqueName: \"kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5\") pod \"payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.624083 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.620595 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d06633-openshift-default-dockercfg-lk8zc\"" Nov 01 14:59:08.707265 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.707209 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:08.727235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727050 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psljl\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727235 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727122 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p74p5\" (UniqueName: \"kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5\") pod \"payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.727466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727246 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727300 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727418 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727466 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727450 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.727681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727479 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727509 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727560 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727593 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.727681 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.727658 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.728898 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.728863 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.729373 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.729347 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.730251 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.730199 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.730467 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.730446 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.730673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.730655 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.735364 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.734865 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.738137 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.738113 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.738532 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.738510 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.791328 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.791292 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.792823 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.792797 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p74p5\" (UniqueName: \"kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5\") pod \"payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.792977 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.792952 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psljl\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl\") pod \"e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.824541 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.824478 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" podUID="fe70667d-623d-4e12-b808-bb1ab138dab8" containerName="payload-processing" containerID="cri-o://56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d" gracePeriod=25 Nov 01 14:59:08.862736 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.862681 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:08.873545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.873522 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:08.921358 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.920629 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:08.923454 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:08.923427 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" podUID="ba9c832a-f5ed-4fc1-9e17-f41f9331d982" containerName="payload-pre-processing" containerID="cri-o://fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375" gracePeriod=25 Nov 01 14:59:09.119304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.118695 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:59:09.176604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.176575 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:09.195199 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.195173 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:59:09.246275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.242775 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4\" (UniqueName: \"kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4\") pod \"fe70667d-623d-4e12-b808-bb1ab138dab8\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " Nov 01 14:59:09.246275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.242977 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume\") pod \"fe70667d-623d-4e12-b808-bb1ab138dab8\" (UID: \"fe70667d-623d-4e12-b808-bb1ab138dab8\") " Nov 01 14:59:09.246275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.244113 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume" pod "fe70667d-623d-4e12-b808-bb1ab138dab8" (UID: "fe70667d-623d-4e12-b808-bb1ab138dab8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:59:09.249433 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.249398 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4" pod "fe70667d-623d-4e12-b808-bb1ab138dab8" (UID: "fe70667d-623d-4e12-b808-bb1ab138dab8"). InnerVolumeSpecName "kube-api-access-6wms4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:09.255048 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.255020 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l"] Nov 01 14:59:09.299737 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:59:09.298647 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80934e4d_60a9_410d_9e9d_8ac3320fa74d.slice/crio-c9ca166edc34362c09c6eef17003a6a622565bcac133d018d7e74b2a391439bc WatchSource:0}: Error finding container c9ca166edc34362c09c6eef17003a6a622565bcac133d018d7e74b2a391439bc: Status 404 returned error can't find the container with id c9ca166edc34362c09c6eef17003a6a622565bcac133d018d7e74b2a391439bc Nov 01 14:59:09.300703 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.300670 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t"] Nov 01 14:59:09.301907 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.301836 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:09.302039 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.301971 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:09.302124 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.302067 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:09.344428 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.344402 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume\") pod \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " Nov 01 14:59:09.344593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.344501 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m\" (UniqueName: \"kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m\") pod \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\" (UID: \"ba9c832a-f5ed-4fc1-9e17-f41f9331d982\") " Nov 01 14:59:09.344857 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.344831 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume" pod "ba9c832a-f5ed-4fc1-9e17-f41f9331d982" (UID: "ba9c832a-f5ed-4fc1-9e17-f41f9331d982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:59:09.345129 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.345084 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe70667d-623d-4e12-b808-bb1ab138dab8-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:09.345244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.345135 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:09.345244 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.345152 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wms4\" (UniqueName: \"kubernetes.io/projected/fe70667d-623d-4e12-b808-bb1ab138dab8-kube-api-access-6wms4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:09.348156 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.348132 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m" pod "ba9c832a-f5ed-4fc1-9e17-f41f9331d982" (UID: "ba9c832a-f5ed-4fc1-9e17-f41f9331d982"). InnerVolumeSpecName "kube-api-access-r9b9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:09.423287 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423248 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd"] Nov 01 14:59:09.423829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423807 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fe70667d-623d-4e12-b808-bb1ab138dab8" containerName="payload-processing" Nov 01 14:59:09.423829 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423822 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe70667d-623d-4e12-b808-bb1ab138dab8" containerName="payload-processing" Nov 01 14:59:09.423945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423848 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ba9c832a-f5ed-4fc1-9e17-f41f9331d982" containerName="payload-pre-processing" Nov 01 14:59:09.423945 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423854 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9c832a-f5ed-4fc1-9e17-f41f9331d982" containerName="payload-pre-processing" Nov 01 14:59:09.424072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423950 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe70667d-623d-4e12-b808-bb1ab138dab8" containerName="payload-processing" Nov 01 14:59:09.424072 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.423962 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9c832a-f5ed-4fc1-9e17-f41f9331d982" containerName="payload-pre-processing" Nov 01 14:59:09.427610 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.427591 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.446607 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.446577 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9b9m\" (UniqueName: \"kubernetes.io/projected/ba9c832a-f5ed-4fc1-9e17-f41f9331d982-kube-api-access-r9b9m\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:09.480202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.480163 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0c9d0a-openshift-default-dockercfg-2zw45\"" Nov 01 14:59:09.486556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.486527 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd"] Nov 01 14:59:09.501130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.501090 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" event={"ID":"000947b6-3732-41f9-ba1b-86f344ac0d2a","Type":"ContainerStarted","Data":"ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c"} Nov 01 14:59:09.501349 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.501151 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" event={"ID":"000947b6-3732-41f9-ba1b-86f344ac0d2a","Type":"ContainerStarted","Data":"0ce8f1ffd60f5ac20868291c868477cd9a8304cb8e7f6df16dc3088dacf920ea"} Nov 01 14:59:09.501862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.501837 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:09.503770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.503745 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" event={"ID":"80934e4d-60a9-410d-9e9d-8ac3320fa74d","Type":"ContainerStarted","Data":"bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77"} Nov 01 14:59:09.503890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.503791 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" event={"ID":"80934e4d-60a9-410d-9e9d-8ac3320fa74d","Type":"ContainerStarted","Data":"c9ca166edc34362c09c6eef17003a6a622565bcac133d018d7e74b2a391439bc"} Nov 01 14:59:09.505726 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.505698 5473 generic.go:356] "Generic (PLEG): container finished" podID="ba9c832a-f5ed-4fc1-9e17-f41f9331d982" containerID="fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375" exitCode=0 Nov 01 14:59:09.505860 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.505844 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" Nov 01 14:59:09.505956 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.505933 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" event={"ID":"ba9c832a-f5ed-4fc1-9e17-f41f9331d982","Type":"ContainerDied","Data":"fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375"} Nov 01 14:59:09.506055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.505987 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t" event={"ID":"ba9c832a-f5ed-4fc1-9e17-f41f9331d982","Type":"ContainerDied","Data":"5e72de5c4e03567eed0dd6a2447509fdb16466d327ee2c6929f09acecdf96e62"} Nov 01 14:59:09.506055 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.506020 5473 scope.go:122] "RemoveContainer" containerID="fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375" Nov 01 14:59:09.508041 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.508017 5473 generic.go:356] "Generic (PLEG): container finished" podID="fe70667d-623d-4e12-b808-bb1ab138dab8" containerID="56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d" exitCode=0 Nov 01 14:59:09.508130 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.508087 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" Nov 01 14:59:09.508203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.508131 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" event={"ID":"fe70667d-623d-4e12-b808-bb1ab138dab8","Type":"ContainerDied","Data":"56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d"} Nov 01 14:59:09.508203 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.508169 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj" event={"ID":"fe70667d-623d-4e12-b808-bb1ab138dab8","Type":"ContainerDied","Data":"9e8a2901104404fb6bb41c2cfb7990f7c15cef54d02bbb799b95542878ba1c35"} Nov 01 14:59:09.522418 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.522379 5473 scope.go:122] "RemoveContainer" containerID="fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375" Nov 01 14:59:09.523089 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:09.523062 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375\": container with ID starting with fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375 not found: ID does not exist" containerID="fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375" Nov 01 14:59:09.523189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.523101 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375"} err="failed to get container status \"fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375\": rpc error: code = NotFound desc = could not find container \"fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375\": container with ID starting with fcb78a02bdc924ee05bcc733d9f839b07679ce6ac9b12213ca0db1cd99492375 not found: ID does not exist" Nov 01 14:59:09.523189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.523129 5473 scope.go:122] "RemoveContainer" containerID="56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d" Nov 01 14:59:09.537336 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.537310 5473 scope.go:122] "RemoveContainer" containerID="56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d" Nov 01 14:59:09.537925 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:09.537898 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d\": container with ID starting with 56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d not found: ID does not exist" containerID="56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d" Nov 01 14:59:09.538026 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.537935 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d"} err="failed to get container status \"56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d\": rpc error: code = NotFound desc = could not find container \"56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d\": container with ID starting with 56c45d3d24892a83da001b9d3773d5da93dd8df01b2318bea1d38341c64ac60d not found: ID does not exist" Nov 01 14:59:09.541845 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.541818 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:09.548004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.547966 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548182 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548235 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548399 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548368 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548464 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548417 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548538 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548516 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548635 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548619 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548695 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548659 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqjc\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.548752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.548731 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650161 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650263 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650208 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650279 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650457 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650504 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqjc\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650545 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650536 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650617 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650790 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650737 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650817 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650832 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.650913 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.650895 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.651202 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.651179 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.651311 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.651239 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.651477 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.651456 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.653474 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.653449 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.653578 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.653564 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.668365 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.668300 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" podStartSLOduration=2.6682803379999998 podStartE2EDuration="2.668280338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:09.661767857 +0000 UTC m=+2488.719919729" watchObservedRunningTime="2026-11-01 14:59:09.668280338 +0000 UTC m=+2488.726432190" Nov 01 14:59:09.713762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.713729 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.714021 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.713940 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqjc\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc\") pod \"e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.742802 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.742772 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:09.795942 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.795907 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t"] Nov 01 14:59:09.800579 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.800549 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5223a8-f4f8f6695-z4f8t"] Nov 01 14:59:09.823593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.823532 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:09.873935 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.873754 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:09.900858 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:09.900754 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podStartSLOduration=1.90073405 podStartE2EDuration="1.90073405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:09.89544644 +0000 UTC m=+2488.953598314" watchObservedRunningTime="2026-11-01 14:59:09.90073405 +0000 UTC m=+2488.958885904" Nov 01 14:59:10.009903 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.009864 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj"] Nov 01 14:59:10.017118 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.017081 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5223a8-574654c9f-4j5fj"] Nov 01 14:59:10.161366 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:59:10.161333 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ca75c2_3332_4ab6_b5b5_a3f1c79ee72b.slice/crio-9b12f402efd0fb585ee8b0a3bb1e4c0a3224037c6a4ee21c88eae78ecd007850 WatchSource:0}: Error finding container 9b12f402efd0fb585ee8b0a3bb1e4c0a3224037c6a4ee21c88eae78ecd007850: Status 404 returned error can't find the container with id 9b12f402efd0fb585ee8b0a3bb1e4c0a3224037c6a4ee21c88eae78ecd007850 Nov 01 14:59:10.164261 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.164191 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:10.164362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.164308 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:10.164362 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.164356 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:10.165719 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.165695 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd"] Nov 01 14:59:10.302189 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.302142 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:10.489628 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.489537 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:10.520746 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.520706 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" event={"ID":"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b","Type":"ContainerStarted","Data":"4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00"} Nov 01 14:59:10.520960 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.520775 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" event={"ID":"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b","Type":"ContainerStarted","Data":"9b12f402efd0fb585ee8b0a3bb1e4c0a3224037c6a4ee21c88eae78ecd007850"} Nov 01 14:59:10.641165 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.641111 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podStartSLOduration=1.64109572 podStartE2EDuration="1.64109572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:10.637532131 +0000 UTC m=+2489.695683984" watchObservedRunningTime="2026-11-01 14:59:10.64109572 +0000 UTC m=+2489.699247570" Nov 01 14:59:10.743592 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.743499 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:10.880400 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:10.880370 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:11.084188 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:11.084151 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9c832a-f5ed-4fc1-9e17-f41f9331d982" path="/var/lib/kubelet/pods/ba9c832a-f5ed-4fc1-9e17-f41f9331d982/volumes" Nov 01 14:59:11.084717 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:11.084694 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe70667d-623d-4e12-b808-bb1ab138dab8" path="/var/lib/kubelet/pods/fe70667d-623d-4e12-b808-bb1ab138dab8/volumes" Nov 01 14:59:11.530279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:11.530241 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:11.531255 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:11.531202 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 14:59:11.748890 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:11.748858 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:12.436997 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:12.436957 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:12.538332 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:12.538295 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:12.539227 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:12.539198 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 14:59:12.608798 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:12.608768 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:12.794581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:12.794548 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:12.965063 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:12.965030 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:13.349832 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:13.349795 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:13.543807 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:13.543774 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:13.717734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:13.717653 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:13.838473 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:13.838446 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 14:59:13.838659 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:13.838549 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 14:59:14.053593 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:14.053563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:15.671011 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:15.670977 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:15.826801 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:15.826771 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:16.088934 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:16.088901 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:18.797461 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:18.797415 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:20.162941 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:20.162909 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:20.339563 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:20.339527 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:21.533420 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:21.533389 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 14:59:22.349787 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.349755 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf"] Nov 01 14:59:22.355299 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.355282 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.407701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.407675 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 14:59:22.407701 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.407699 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 14:59:22.407911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.407699 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 14:59:22.407911 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.407811 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lmbst\"" Nov 01 14:59:22.427622 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.427589 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.427753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.427651 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.427753 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.427718 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.427849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.427760 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.427849 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.427787 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.427950 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.427860 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.428028 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.428007 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hswz\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.463179 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.463145 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf"] Nov 01 14:59:22.529577 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529542 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.529739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529612 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.529739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529640 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.529739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529678 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.529739 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529715 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.529989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529740 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.529989 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.529809 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hswz\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.530583 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.530557 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.532120 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.532088 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.532266 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.532171 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.532581 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.532561 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.532629 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.532600 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.586345 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.586314 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.586725 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.586644 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hswz\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rq7xf\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:22.667850 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:22.667761 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:23.090003 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.089970 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf"] Nov 01 14:59:23.120546 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:59:23.120518 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98b3fa4_5995_498c_9190_e87eb66aeb4a.slice/crio-323558dd5ed32cd67457fb0c287b87d70e50ee09e8fde16c7f2ff6bf9f6f7c53 WatchSource:0}: Error finding container 323558dd5ed32cd67457fb0c287b87d70e50ee09e8fde16c7f2ff6bf9f6f7c53: Status 404 returned error can't find the container with id 323558dd5ed32cd67457fb0c287b87d70e50ee09e8fde16c7f2ff6bf9f6f7c53 Nov 01 14:59:23.122995 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.122962 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:23.123071 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.123024 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 14:59:23.328521 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.328486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:23.626504 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.626459 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" event={"ID":"f98b3fa4-5995-498c-9190-e87eb66aeb4a","Type":"ContainerStarted","Data":"5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3"} Nov 01 14:59:23.626888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.626514 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" event={"ID":"f98b3fa4-5995-498c-9190-e87eb66aeb4a","Type":"ContainerStarted","Data":"323558dd5ed32cd67457fb0c287b87d70e50ee09e8fde16c7f2ff6bf9f6f7c53"} Nov 01 14:59:23.626888 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.626699 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:23.744751 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:23.744693 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" podStartSLOduration=1.744673428 podStartE2EDuration="1.744673428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:23.742863615 +0000 UTC m=+2502.801015465" watchObservedRunningTime="2026-11-01 14:59:23.744673428 +0000 UTC m=+2502.802825286" Nov 01 14:59:24.049673 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:24.049637 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp"] Nov 01 14:59:24.053524 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:24.053501 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d8b148d-q5wgp"] Nov 01 14:59:24.640678 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:24.640646 5473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rq7xf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 14:59:24.641140 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:24.641112 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" podUID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 14:59:25.083355 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:25.083312 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc5bbfc-3085-48f7-8230-b3433c507cd3" path="/var/lib/kubelet/pods/2bc5bbfc-3085-48f7-8230-b3433c507cd3/volumes" Nov 01 14:59:27.210321 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:27.210285 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:27.640056 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:27.640023 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 14:59:28.789584 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:28.789541 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:28.960671 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:28.960612 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:29.144294 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:29.143702 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:30.188757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:30.188373 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 14:59:30.188757 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:30.188440 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 14:59:31.119084 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:31.119032 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 14:59:33.542383 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:33.542349 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:33.858099 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:33.858009 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:34.015163 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:34.015130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:35.365023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:35.364988 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:35.843542 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:35.843506 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w"] Nov 01 14:59:35.843789 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:35.843759 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" containerID="cri-o://0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407" gracePeriod=30 Nov 01 14:59:35.868329 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:35.868299 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:38.261103 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:38.261063 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:38.462302 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:38.462239 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:38.618470 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:38.618436 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:41.122519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.122465 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:59:41.207132 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.207097 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.213818 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.208187 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:41.214090 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214053 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214122 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214179 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214289 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214256 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214324 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214495 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214456 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214503 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.214617 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.214575 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert\") pod \"ff0c815e-4363-4ce1-ad46-bccbe276941b\" (UID: \"ff0c815e-4363-4ce1-ad46-bccbe276941b\") " Nov 01 14:59:41.215240 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.215186 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:41.215519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.215498 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.215519 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.215522 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.215667 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.215583 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:41.219270 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.219243 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:41.219481 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.219457 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:59:41.219734 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.219662 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "kube-api-access-zrjhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:41.221337 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.221309 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:41.222613 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.222590 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 14:59:41.222704 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.222636 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy" pod "ff0c815e-4363-4ce1-ad46-bccbe276941b" (UID: "ff0c815e-4363-4ce1-ad46-bccbe276941b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 14:59:41.316770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316731 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.316770 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316772 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.316984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316788 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.316984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316801 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.316984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316815 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrjhc\" (UniqueName: \"kubernetes.io/projected/ff0c815e-4363-4ce1-ad46-bccbe276941b-kube-api-access-zrjhc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.316984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316830 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0c815e-4363-4ce1-ad46-bccbe276941b-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.316984 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.316843 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0c815e-4363-4ce1-ad46-bccbe276941b-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:41.792175 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.792144 5473 generic.go:356] "Generic (PLEG): container finished" podID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerID="0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407" exitCode=0 Nov 01 14:59:41.792368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.792191 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" Nov 01 14:59:41.792368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.792250 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" event={"ID":"ff0c815e-4363-4ce1-ad46-bccbe276941b","Type":"ContainerDied","Data":"0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407"} Nov 01 14:59:41.792368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.792308 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w" event={"ID":"ff0c815e-4363-4ce1-ad46-bccbe276941b","Type":"ContainerDied","Data":"13068b746bdb5fe3502c6cd83f0c449863531addb4fb7485442b427297f949dd"} Nov 01 14:59:41.792368 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.792330 5473 scope.go:122] "RemoveContainer" containerID="0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407" Nov 01 14:59:41.803445 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.803426 5473 scope.go:122] "RemoveContainer" containerID="4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1" Nov 01 14:59:41.813238 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.813195 5473 scope.go:122] "RemoveContainer" containerID="0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407" Nov 01 14:59:41.813561 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:41.813539 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407\": container with ID starting with 0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407 not found: ID does not exist" containerID="0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407" Nov 01 14:59:41.813604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.813570 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407"} err="failed to get container status \"0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407\": rpc error: code = NotFound desc = could not find container \"0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407\": container with ID starting with 0465758c0c73023bef5f44de21786de52ca2b2a0b2204b97e524ba2a7df4f407 not found: ID does not exist" Nov 01 14:59:41.813604 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.813587 5473 scope.go:122] "RemoveContainer" containerID="4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1" Nov 01 14:59:41.813849 ip-10-0-131-109 kubenswrapper[5473]: E1101 14:59:41.813833 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1\": container with ID starting with 4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1 not found: ID does not exist" containerID="4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1" Nov 01 14:59:41.813886 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:41.813856 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1"} err="failed to get container status \"4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1\": rpc error: code = NotFound desc = could not find container \"4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1\": container with ID starting with 4a1ac1322c68bf996867aec1e4ce1a5d76aa6a5615d3f055b729ea1b9f11f4c1 not found: ID does not exist" Nov 01 14:59:42.009145 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:42.009113 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w"] Nov 01 14:59:42.059211 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:42.059137 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5223a8-openshift-default-79497fb9cc-q5p8w"] Nov 01 14:59:42.063004 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:42.062988 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw"] Nov 01 14:59:42.066317 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:42.066298 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54d3a927-nrlgw"] Nov 01 14:59:43.081756 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:43.081723 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a53bcf-a11f-4f27-8a0d-4564ab53746d" path="/var/lib/kubelet/pods/e4a53bcf-a11f-4f27-8a0d-4564ab53746d/volumes" Nov 01 14:59:43.082243 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:43.082227 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" path="/var/lib/kubelet/pods/ff0c815e-4363-4ce1-ad46-bccbe276941b/volumes" Nov 01 14:59:43.283551 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:43.283522 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:47.361078 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361044 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk"] Nov 01 14:59:47.361620 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361605 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" Nov 01 14:59:47.361668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361622 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" Nov 01 14:59:47.361668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361648 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" Nov 01 14:59:47.361668 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361656 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" Nov 01 14:59:47.361780 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361770 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" Nov 01 14:59:47.361817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.361783 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0c815e-4363-4ce1-ad46-bccbe276941b" containerName="istio-proxy" Nov 01 14:59:47.367348 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.367332 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.419229 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.419183 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk"] Nov 01 14:59:47.462803 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.462775 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 14:59:47.462986 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.462902 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 14:59:47.496731 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.496698 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddzk\" (UniqueName: \"kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.496862 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.496743 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.496938 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.496920 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.598275 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.598207 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddzk\" (UniqueName: \"kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.598483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.598296 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.598483 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.598408 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.598980 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.598956 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.600752 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.600725 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.654741 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.654675 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddzk\" (UniqueName: \"kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk\") pod \"maas-api-revoke-sub-caebccb3-pcqhk\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.681556 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.681528 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:47.831087 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:47.831056 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:48.050009 ip-10-0-131-109 kubenswrapper[5473]: W1101 14:59:48.049969 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c26cdb_a883_4683_bcf6_3d268794fd49.slice/crio-aff856c9427d22bee31a9073b5d9e63d2acd798fe243447ec499817d4b5e3bac WatchSource:0}: Error finding container aff856c9427d22bee31a9073b5d9e63d2acd798fe243447ec499817d4b5e3bac: Status 404 returned error can't find the container with id aff856c9427d22bee31a9073b5d9e63d2acd798fe243447ec499817d4b5e3bac Nov 01 14:59:48.054397 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:48.054372 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk"] Nov 01 14:59:48.854279 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:48.854240 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" event={"ID":"39c26cdb-a883-4683-bcf6-3d268794fd49","Type":"ContainerStarted","Data":"ec88fb90ae8ad3bcd0e7d6a1f686f61dc6f4103fdc521b45839409fdb4e667d7"} Nov 01 14:59:48.854749 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:48.854292 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" event={"ID":"39c26cdb-a883-4683-bcf6-3d268794fd49","Type":"ContainerStarted","Data":"aff856c9427d22bee31a9073b5d9e63d2acd798fe243447ec499817d4b5e3bac"} Nov 01 14:59:48.993762 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:48.993712 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" podStartSLOduration=1.993698339 podStartE2EDuration="1.993698339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 14:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 14:59:48.991481287 +0000 UTC m=+2528.049633136" watchObservedRunningTime="2026-11-01 14:59:48.993698339 +0000 UTC m=+2528.051850189" Nov 01 14:59:49.863866 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:49.863833 5473 generic.go:356] "Generic (PLEG): container finished" podID="39c26cdb-a883-4683-bcf6-3d268794fd49" containerID="ec88fb90ae8ad3bcd0e7d6a1f686f61dc6f4103fdc521b45839409fdb4e667d7" exitCode=0 Nov 01 14:59:49.864304 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:49.863910 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" event={"ID":"39c26cdb-a883-4683-bcf6-3d268794fd49","Type":"ContainerDied","Data":"ec88fb90ae8ad3bcd0e7d6a1f686f61dc6f4103fdc521b45839409fdb4e667d7"} Nov 01 14:59:51.027694 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.027666 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:51.034884 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.034863 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca\") pod \"39c26cdb-a883-4683-bcf6-3d268794fd49\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " Nov 01 14:59:51.034999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.034897 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca\") pod \"39c26cdb-a883-4683-bcf6-3d268794fd49\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " Nov 01 14:59:51.034999 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.034976 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk\" (UniqueName: \"kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk\") pod \"39c26cdb-a883-4683-bcf6-3d268794fd49\" (UID: \"39c26cdb-a883-4683-bcf6-3d268794fd49\") " Nov 01 14:59:51.035331 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.035309 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca" pod "39c26cdb-a883-4683-bcf6-3d268794fd49" (UID: "39c26cdb-a883-4683-bcf6-3d268794fd49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 14:59:51.036990 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.036969 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk" pod "39c26cdb-a883-4683-bcf6-3d268794fd49" (UID: "39c26cdb-a883-4683-bcf6-3d268794fd49"). InnerVolumeSpecName "kube-api-access-qddzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 14:59:51.037064 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.037002 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca" pod "39c26cdb-a883-4683-bcf6-3d268794fd49" (UID: "39c26cdb-a883-4683-bcf6-3d268794fd49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 14:59:51.136655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.136593 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:51.136655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.136626 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c26cdb-a883-4683-bcf6-3d268794fd49-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:51.136655 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.136641 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddzk\" (UniqueName: \"kubernetes.io/projected/39c26cdb-a883-4683-bcf6-3d268794fd49-kube-api-access-qddzk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 14:59:51.880817 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.880780 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" event={"ID":"39c26cdb-a883-4683-bcf6-3d268794fd49","Type":"ContainerDied","Data":"aff856c9427d22bee31a9073b5d9e63d2acd798fe243447ec499817d4b5e3bac"} Nov 01 14:59:51.880996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.880839 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aff856c9427d22bee31a9073b5d9e63d2acd798fe243447ec499817d4b5e3bac" Nov 01 14:59:51.880996 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:51.880788 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk" Nov 01 14:59:52.249979 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:52.249902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:52.410568 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:52.410533 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:52.575745 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:52.575715 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:56.466424 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:56.466386 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:56.870023 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:56.869987 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:57.032319 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:57.032278 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:57.193633 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:57.193550 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 14:59:59.050257 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:59.050210 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22"] Nov 01 14:59:59.091510 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:59.091476 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae65426-jkm22"] Nov 01 14:59:59.359283 ip-10-0-131-109 kubenswrapper[5473]: I1101 14:59:59.359172 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:00.023623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.023587 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:00.164199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.164165 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4"] Nov 01 15:00:00.165034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.165009 5473 cpu_manager.go:414] "containerMap: removing container" podUID="39c26cdb-a883-4683-bcf6-3d268794fd49" containerName="revoke-keys" Nov 01 15:00:00.165034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.165034 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c26cdb-a883-4683-bcf6-3d268794fd49" containerName="revoke-keys" Nov 01 15:00:00.165206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.165145 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c26cdb-a883-4683-bcf6-3d268794fd49" containerName="revoke-keys" Nov 01 15:00:00.169289 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.169271 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.174168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.174151 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2"] Nov 01 15:00:00.178536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.178518 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.179169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.179147 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm"] Nov 01 15:00:00.184022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.184000 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww"] Nov 01 15:00:00.184130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.184102 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.188473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.188450 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.213099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.213064 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428"] Nov 01 15:00:00.217836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.217815 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww"] Nov 01 15:00:00.217935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.217919 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.225858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.225834 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4"] Nov 01 15:00:00.226257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.225949 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.235592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.235565 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2"] Nov 01 15:00:00.248858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.248825 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.248980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.248878 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzsfq\" (UniqueName: \"kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.248980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.248970 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.249066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.249011 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvds\" (UniqueName: \"kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.249199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.249177 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mncfc\" (UniqueName: \"kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.249293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.249273 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9cdm\" (UniqueName: \"kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.249348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.249308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.249446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.249429 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.263462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.263429 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:00:00.265866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.265839 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm"] Nov 01 15:00:00.273071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.273048 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428"] Nov 01 15:00:00.282301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.279820 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww"] Nov 01 15:00:00.287768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.287742 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww"] Nov 01 15:00:00.350649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350613 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892420-t6428\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.350840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350658 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.350840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350684 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmkl\" (UniqueName: \"kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.350840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350824 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzsfq\" (UniqueName: \"kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.350981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350849 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zph\" (UniqueName: \"kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph\") pod \"maas-api-key-cleanup-29892420-t6428\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.350981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350873 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.350981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350924 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvds\" (UniqueName: \"kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.350981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.350960 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.351195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.351097 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mncfc\" (UniqueName: \"kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.351195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.351144 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9cdm\" (UniqueName: \"kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.351195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.351171 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.351386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.351310 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.351536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.351511 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.351769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.351750 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.352042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.352022 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.352186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.352166 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.380761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.380721 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:00.410496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.410467 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzsfq\" (UniqueName: \"kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.410693 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.410673 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9cdm\" (UniqueName: \"kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.410764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.410743 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvds\" (UniqueName: \"kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.410837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.410791 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mncfc\" (UniqueName: \"kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.453128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.453101 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892420-t6428\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.453259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.453150 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmkl\" (UniqueName: \"kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.453259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.453200 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zph\" (UniqueName: \"kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph\") pod \"maas-api-key-cleanup-29892420-t6428\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.453403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.453291 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.453792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.453772 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892420-t6428\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.453895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.453877 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.485931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.485911 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:00.495573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.495555 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:00.505189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.505166 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:00.512188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.511955 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:00.513168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.513145 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmkl\" (UniqueName: \"kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.514263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.514202 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zph\" (UniqueName: \"kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph\") pod \"maas-api-key-cleanup-29892420-t6428\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.537252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.537016 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:00.545920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.545705 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:00.876293 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:00.876259 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode998b9dd_027d_42bc_a948_93ed4ae6c0e8.slice/crio-39e3c18e389083336488399c985239ec2aa5ac6fb27ac8bfcd95af1a19c0c3fa WatchSource:0}: Error finding container 39e3c18e389083336488399c985239ec2aa5ac6fb27ac8bfcd95af1a19c0c3fa: Status 404 returned error can't find the container with id 39e3c18e389083336488399c985239ec2aa5ac6fb27ac8bfcd95af1a19c0c3fa Nov 01 15:00:00.881242 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.881194 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4"] Nov 01 15:00:00.925958 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:00.925920 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5269d12_6b1e_4153_b249_472a15a24d19.slice/crio-87b219551fb82c5f119780a79e52a9881efd34ac341a83458583b405fb4083aa WatchSource:0}: Error finding container 87b219551fb82c5f119780a79e52a9881efd34ac341a83458583b405fb4083aa: Status 404 returned error can't find the container with id 87b219551fb82c5f119780a79e52a9881efd34ac341a83458583b405fb4083aa Nov 01 15:00:00.931006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.930978 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm"] Nov 01 15:00:00.973334 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:00.973308 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3440bb1b_53b2_44ec_91b4_841dd3d7ec22.slice/crio-610e0d79f82b80adbe71a3e72bf67a7fcbed07b5ff7ae4ea779edc54f70e8e08 WatchSource:0}: Error finding container 610e0d79f82b80adbe71a3e72bf67a7fcbed07b5ff7ae4ea779edc54f70e8e08: Status 404 returned error can't find the container with id 610e0d79f82b80adbe71a3e72bf67a7fcbed07b5ff7ae4ea779edc54f70e8e08 Nov 01 15:00:00.975384 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:00.975364 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a82f9f_afce_4221_80fe_f52cc264eee0.slice/crio-0602e89f404504533f6952ef2dcdc837a9b30657d216324423a9ac91f709ac4b WatchSource:0}: Error finding container 0602e89f404504533f6952ef2dcdc837a9b30657d216324423a9ac91f709ac4b: Status 404 returned error can't find the container with id 0602e89f404504533f6952ef2dcdc837a9b30657d216324423a9ac91f709ac4b Nov 01 15:00:00.979375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.979158 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww"] Nov 01 15:00:00.989665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:00.989635 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428"] Nov 01 15:00:01.004486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.004460 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww"] Nov 01 15:00:01.009502 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:01.009475 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode617dc74_0ae2_4e1f_9eb5_6a9a0a02181f.slice/crio-6cc5b4033e646f3236b1463c9a926fd1ab001d0c9cf044148344fad18c6a2425 WatchSource:0}: Error finding container 6cc5b4033e646f3236b1463c9a926fd1ab001d0c9cf044148344fad18c6a2425: Status 404 returned error can't find the container with id 6cc5b4033e646f3236b1463c9a926fd1ab001d0c9cf044148344fad18c6a2425 Nov 01 15:00:01.030085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.030052 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" event={"ID":"3440bb1b-53b2-44ec-91b4-841dd3d7ec22","Type":"ContainerStarted","Data":"610e0d79f82b80adbe71a3e72bf67a7fcbed07b5ff7ae4ea779edc54f70e8e08"} Nov 01 15:00:01.031667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.031634 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" event={"ID":"e998b9dd-027d-42bc-a948-93ed4ae6c0e8","Type":"ContainerStarted","Data":"39e3c18e389083336488399c985239ec2aa5ac6fb27ac8bfcd95af1a19c0c3fa"} Nov 01 15:00:01.033092 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.033065 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" event={"ID":"85a82f9f-afce-4221-80fe-f52cc264eee0","Type":"ContainerStarted","Data":"0602e89f404504533f6952ef2dcdc837a9b30657d216324423a9ac91f709ac4b"} Nov 01 15:00:01.035791 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.035501 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" event={"ID":"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f","Type":"ContainerStarted","Data":"6cc5b4033e646f3236b1463c9a926fd1ab001d0c9cf044148344fad18c6a2425"} Nov 01 15:00:01.037137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.037103 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerStarted","Data":"87b219551fb82c5f119780a79e52a9881efd34ac341a83458583b405fb4083aa"} Nov 01 15:00:01.040814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.040793 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2"] Nov 01 15:00:01.083111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.083086 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1a3d2b-7856-4e02-bee1-b89ac1660229" path="/var/lib/kubelet/pods/db1a3d2b-7856-4e02-bee1-b89ac1660229/volumes" Nov 01 15:00:01.107892 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:01.107862 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:00:01.739453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:01.739423 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:02.047191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.047164 5473 generic.go:356] "Generic (PLEG): container finished" podID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerID="3411d9a9ba7a63abc266095cfd28f4a80b1c196f160f8bc0817a1e810b71dbf7" exitCode=60 Nov 01 15:00:02.047351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.047260 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" event={"ID":"85a82f9f-afce-4221-80fe-f52cc264eee0","Type":"ContainerDied","Data":"3411d9a9ba7a63abc266095cfd28f4a80b1c196f160f8bc0817a1e810b71dbf7"} Nov 01 15:00:02.047575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.047561 5473 scope.go:122] "RemoveContainer" containerID="3411d9a9ba7a63abc266095cfd28f4a80b1c196f160f8bc0817a1e810b71dbf7" Nov 01 15:00:02.049107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.049084 5473 generic.go:356] "Generic (PLEG): container finished" podID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerID="a811d132a027d40ff69c9e03951295230d94173b0c6222aa676072e44e3cc63f" exitCode=60 Nov 01 15:00:02.049193 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.049118 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" event={"ID":"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f","Type":"ContainerDied","Data":"a811d132a027d40ff69c9e03951295230d94173b0c6222aa676072e44e3cc63f"} Nov 01 15:00:02.049421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.049405 5473 scope.go:122] "RemoveContainer" containerID="a811d132a027d40ff69c9e03951295230d94173b0c6222aa676072e44e3cc63f" Nov 01 15:00:02.050972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.050949 5473 generic.go:356] "Generic (PLEG): container finished" podID="e5269d12-6b1e-4153-b249-472a15a24d19" containerID="aab642e90886f97e186e36fd30342e332d4bca1b93746d49fcf0b2afc636beed" exitCode=60 Nov 01 15:00:02.051110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.051028 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerDied","Data":"aab642e90886f97e186e36fd30342e332d4bca1b93746d49fcf0b2afc636beed"} Nov 01 15:00:02.051332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.051310 5473 scope.go:122] "RemoveContainer" containerID="aab642e90886f97e186e36fd30342e332d4bca1b93746d49fcf0b2afc636beed" Nov 01 15:00:02.052996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.052977 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerStarted","Data":"0567ffc6a5cb3d4ccdd02bb80736abd1c7c34cacda1e55dbe98be7720b36c8af"} Nov 01 15:00:02.053059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.053019 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerStarted","Data":"9b85ee14b21336ceeeb91c57b86dc110da02c431bd64bc0c148802548ab97653"} Nov 01 15:00:02.054554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.054534 5473 generic.go:356] "Generic (PLEG): container finished" podID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerID="24c252ac1019265be71ea19ecf110ba232c47b8f4a3a928616eb2b9e5beb4028" exitCode=60 Nov 01 15:00:02.054625 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.054606 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" event={"ID":"3440bb1b-53b2-44ec-91b4-841dd3d7ec22","Type":"ContainerDied","Data":"24c252ac1019265be71ea19ecf110ba232c47b8f4a3a928616eb2b9e5beb4028"} Nov 01 15:00:02.054832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.054818 5473 scope.go:122] "RemoveContainer" containerID="24c252ac1019265be71ea19ecf110ba232c47b8f4a3a928616eb2b9e5beb4028" Nov 01 15:00:02.056369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.056350 5473 generic.go:356] "Generic (PLEG): container finished" podID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerID="2232db0505224c87bd89f359540e443cdeca6511d6724518249af2c25bdbb845" exitCode=60 Nov 01 15:00:02.056471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.056408 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" event={"ID":"e998b9dd-027d-42bc-a948-93ed4ae6c0e8","Type":"ContainerDied","Data":"2232db0505224c87bd89f359540e443cdeca6511d6724518249af2c25bdbb845"} Nov 01 15:00:02.056635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.056620 5473 scope.go:122] "RemoveContainer" containerID="2232db0505224c87bd89f359540e443cdeca6511d6724518249af2c25bdbb845" Nov 01 15:00:02.729525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:02.729464 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" podStartSLOduration=2.72944595 podStartE2EDuration="2.72944595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:02.724865 +0000 UTC m=+2541.783016851" watchObservedRunningTime="2026-11-01 15:00:02.72944595 +0000 UTC m=+2541.787597801" Nov 01 15:00:03.065542 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.065507 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" event={"ID":"e998b9dd-027d-42bc-a948-93ed4ae6c0e8","Type":"ContainerStarted","Data":"e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702"} Nov 01 15:00:03.067500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.067476 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" event={"ID":"85a82f9f-afce-4221-80fe-f52cc264eee0","Type":"ContainerStarted","Data":"dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9"} Nov 01 15:00:03.069201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.069179 5473 generic.go:356] "Generic (PLEG): container finished" podID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerID="a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce" exitCode=60 Nov 01 15:00:03.069346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.069294 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" event={"ID":"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f","Type":"ContainerDied","Data":"a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce"} Nov 01 15:00:03.069410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.069353 5473 scope.go:122] "RemoveContainer" containerID="a811d132a027d40ff69c9e03951295230d94173b0c6222aa676072e44e3cc63f" Nov 01 15:00:03.069515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.069492 5473 scope.go:122] "RemoveContainer" containerID="a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce" Nov 01 15:00:03.069788 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:03.069764 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww_odh-ai-gateway-infra(e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" Nov 01 15:00:03.071438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.071419 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerStarted","Data":"8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378"} Nov 01 15:00:03.072988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.072969 5473 generic.go:356] "Generic (PLEG): container finished" podID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerID="0567ffc6a5cb3d4ccdd02bb80736abd1c7c34cacda1e55dbe98be7720b36c8af" exitCode=60 Nov 01 15:00:03.073099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.073036 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerDied","Data":"0567ffc6a5cb3d4ccdd02bb80736abd1c7c34cacda1e55dbe98be7720b36c8af"} Nov 01 15:00:03.073197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.073183 5473 scope.go:122] "RemoveContainer" containerID="0567ffc6a5cb3d4ccdd02bb80736abd1c7c34cacda1e55dbe98be7720b36c8af" Nov 01 15:00:03.082618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.082588 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" event={"ID":"3440bb1b-53b2-44ec-91b4-841dd3d7ec22","Type":"ContainerStarted","Data":"c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3"} Nov 01 15:00:03.181890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.181827 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" podStartSLOduration=3.181812653 podStartE2EDuration="3.181812653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:03.179847077 +0000 UTC m=+2542.237998926" watchObservedRunningTime="2026-11-01 15:00:03.181812653 +0000 UTC m=+2542.239964503" Nov 01 15:00:03.410044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.409995 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" podStartSLOduration=3.409980406 podStartE2EDuration="3.409980406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:03.407640352 +0000 UTC m=+2542.465792207" watchObservedRunningTime="2026-11-01 15:00:03.409980406 +0000 UTC m=+2542.468132255" Nov 01 15:00:03.421003 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:03.420971 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode617dc74_0ae2_4e1f_9eb5_6a9a0a02181f.slice/crio-conmon-a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce.scope\": RecentStats: unable to find data in memory cache]" Nov 01 15:00:03.532413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.532364 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" podStartSLOduration=3.532350475 podStartE2EDuration="3.532350475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:03.53147831 +0000 UTC m=+2542.589630160" watchObservedRunningTime="2026-11-01 15:00:03.532350475 +0000 UTC m=+2542.590502325" Nov 01 15:00:03.656166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:03.656052 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" podStartSLOduration=3.656035192 podStartE2EDuration="3.656035192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:03.653009249 +0000 UTC m=+2542.711161098" watchObservedRunningTime="2026-11-01 15:00:03.656035192 +0000 UTC m=+2542.714187043" Nov 01 15:00:04.085192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.085164 5473 generic.go:356] "Generic (PLEG): container finished" podID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerID="c4394d941efc96a9508d7c697019d0a8d34dbc794ec58533723f85da37e4025b" exitCode=60 Nov 01 15:00:04.085624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.085240 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerDied","Data":"c4394d941efc96a9508d7c697019d0a8d34dbc794ec58533723f85da37e4025b"} Nov 01 15:00:04.085624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.085302 5473 scope.go:122] "RemoveContainer" containerID="0567ffc6a5cb3d4ccdd02bb80736abd1c7c34cacda1e55dbe98be7720b36c8af" Nov 01 15:00:04.085624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.085520 5473 scope.go:122] "RemoveContainer" containerID="c4394d941efc96a9508d7c697019d0a8d34dbc794ec58533723f85da37e4025b" Nov 01 15:00:04.085813 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:04.085783 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2_odh-ai-gateway-infra(7dff3765-181b-45b4-9c5a-358c2c543d2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" Nov 01 15:00:04.087352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.087328 5473 generic.go:356] "Generic (PLEG): container finished" podID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerID="c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3" exitCode=60 Nov 01 15:00:04.087487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.087384 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" event={"ID":"3440bb1b-53b2-44ec-91b4-841dd3d7ec22","Type":"ContainerDied","Data":"c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3"} Nov 01 15:00:04.087721 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.087698 5473 scope.go:122] "RemoveContainer" containerID="c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3" Nov 01 15:00:04.087955 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:04.087932 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww_odh-ai-gateway-infra(3440bb1b-53b2-44ec-91b4-841dd3d7ec22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" Nov 01 15:00:04.089589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.089571 5473 generic.go:356] "Generic (PLEG): container finished" podID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerID="e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702" exitCode=60 Nov 01 15:00:04.089688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.089646 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" event={"ID":"e998b9dd-027d-42bc-a948-93ed4ae6c0e8","Type":"ContainerDied","Data":"e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702"} Nov 01 15:00:04.089909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.089891 5473 scope.go:122] "RemoveContainer" containerID="e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702" Nov 01 15:00:04.090183 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:04.090157 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4_odh-ai-gateway-infra(e998b9dd-027d-42bc-a948-93ed4ae6c0e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" Nov 01 15:00:04.091675 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.091656 5473 generic.go:356] "Generic (PLEG): container finished" podID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerID="dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9" exitCode=60 Nov 01 15:00:04.091762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.091733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" event={"ID":"85a82f9f-afce-4221-80fe-f52cc264eee0","Type":"ContainerDied","Data":"dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9"} Nov 01 15:00:04.091919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.091905 5473 scope.go:122] "RemoveContainer" containerID="dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9" Nov 01 15:00:04.092145 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:04.092123 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892420-t6428_odh-ai-gateway-infra(85a82f9f-afce-4221-80fe-f52cc264eee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" Nov 01 15:00:04.093566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.093547 5473 scope.go:122] "RemoveContainer" containerID="a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce" Nov 01 15:00:04.093759 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:04.093737 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww_odh-ai-gateway-infra(e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" Nov 01 15:00:04.095387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.095365 5473 generic.go:356] "Generic (PLEG): container finished" podID="e5269d12-6b1e-4153-b249-472a15a24d19" containerID="8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378" exitCode=60 Nov 01 15:00:04.095478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.095422 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerDied","Data":"8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378"} Nov 01 15:00:04.095655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.095640 5473 scope.go:122] "RemoveContainer" containerID="8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378" Nov 01 15:00:04.095903 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:04.095882 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm_odh-ai-gateway-infra(e5269d12-6b1e-4153-b249-472a15a24d19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" Nov 01 15:00:04.104735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.104705 5473 scope.go:122] "RemoveContainer" containerID="24c252ac1019265be71ea19ecf110ba232c47b8f4a3a928616eb2b9e5beb4028" Nov 01 15:00:04.117804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.117790 5473 scope.go:122] "RemoveContainer" containerID="2232db0505224c87bd89f359540e443cdeca6511d6724518249af2c25bdbb845" Nov 01 15:00:04.130593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.130575 5473 scope.go:122] "RemoveContainer" containerID="3411d9a9ba7a63abc266095cfd28f4a80b1c196f160f8bc0817a1e810b71dbf7" Nov 01 15:00:04.143372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.143352 5473 scope.go:122] "RemoveContainer" containerID="aab642e90886f97e186e36fd30342e332d4bca1b93746d49fcf0b2afc636beed" Nov 01 15:00:04.526022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.525990 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:04.823426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:04.823330 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:05.009782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:05.009748 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:05.104760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:05.104675 5473 scope.go:122] "RemoveContainer" containerID="e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702" Nov 01 15:00:05.105181 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:05.105035 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4_odh-ai-gateway-infra(e998b9dd-027d-42bc-a948-93ed4ae6c0e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" Nov 01 15:00:05.106421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:05.106400 5473 scope.go:122] "RemoveContainer" containerID="dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9" Nov 01 15:00:05.106692 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:05.106670 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892420-t6428_odh-ai-gateway-infra(85a82f9f-afce-4221-80fe-f52cc264eee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" Nov 01 15:00:05.107965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:05.107945 5473 scope.go:122] "RemoveContainer" containerID="8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378" Nov 01 15:00:05.108250 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:05.108228 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm_odh-ai-gateway-infra(e5269d12-6b1e-4153-b249-472a15a24d19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" Nov 01 15:00:05.109455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:05.109438 5473 scope.go:122] "RemoveContainer" containerID="c4394d941efc96a9508d7c697019d0a8d34dbc794ec58533723f85da37e4025b" Nov 01 15:00:05.109682 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:05.109655 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2_odh-ai-gateway-infra(7dff3765-181b-45b4-9c5a-358c2c543d2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" Nov 01 15:00:05.110993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:05.110971 5473 scope.go:122] "RemoveContainer" containerID="c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3" Nov 01 15:00:05.111149 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:05.111133 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww_odh-ai-gateway-infra(3440bb1b-53b2-44ec-91b4-841dd3d7ec22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" Nov 01 15:00:06.538660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:06.538626 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:08.663086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:08.663048 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:08.829032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:08.829001 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:11.504073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:11.504039 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:13.119442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:13.119412 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4"] Nov 01 15:00:13.119908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:13.119452 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90489d23-vq5b4"] Nov 01 15:00:14.075973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:14.075939 5473 scope.go:122] "RemoveContainer" containerID="dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9" Nov 01 15:00:14.077203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:14.077181 5473 scope.go:122] "RemoveContainer" containerID="8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378" Nov 01 15:00:15.110607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:15.110516 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce37353-56fc-4a07-bdc4-7a41e8c28f1a" path="/var/lib/kubelet/pods/9ce37353-56fc-4a07-bdc4-7a41e8c28f1a/volumes" Nov 01 15:00:15.410880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:15.409186 5473 generic.go:356] "Generic (PLEG): container finished" podID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerID="ca97a479d037195a7236ed324bdbcd4a78aab642697b5b4f3387d914d69bf279" exitCode=60 Nov 01 15:00:15.410880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:15.409365 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" event={"ID":"85a82f9f-afce-4221-80fe-f52cc264eee0","Type":"ContainerDied","Data":"ca97a479d037195a7236ed324bdbcd4a78aab642697b5b4f3387d914d69bf279"} Nov 01 15:00:15.410880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:15.409425 5473 scope.go:122] "RemoveContainer" containerID="dda6e7085098f89bb8ef77ce42cf721f88816da98dcf9df435da5b5f5190fef9" Nov 01 15:00:15.410880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:15.409856 5473 scope.go:122] "RemoveContainer" containerID="ca97a479d037195a7236ed324bdbcd4a78aab642697b5b4f3387d914d69bf279" Nov 01 15:00:15.410880 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:15.410156 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892420-t6428_odh-ai-gateway-infra(85a82f9f-afce-4221-80fe-f52cc264eee0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" Nov 01 15:00:15.414674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:15.414645 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerStarted","Data":"f8ed9b991ab6667b1e03a69f211b723f7505f85df48c5eec4a6bcea80c431a52"} Nov 01 15:00:16.078241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.077918 5473 scope.go:122] "RemoveContainer" containerID="c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3" Nov 01 15:00:16.308716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.308486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:16.485249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.485199 5473 generic.go:356] "Generic (PLEG): container finished" podID="e5269d12-6b1e-4153-b249-472a15a24d19" containerID="f8ed9b991ab6667b1e03a69f211b723f7505f85df48c5eec4a6bcea80c431a52" exitCode=60 Nov 01 15:00:16.485881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.485853 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerDied","Data":"f8ed9b991ab6667b1e03a69f211b723f7505f85df48c5eec4a6bcea80c431a52"} Nov 01 15:00:16.486124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.486089 5473 scope.go:122] "RemoveContainer" containerID="8351ea7eea1b0fed66e233125f820ed899b1aaad55fabbce9b080e8848ffc378" Nov 01 15:00:16.491575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.491557 5473 scope.go:122] "RemoveContainer" containerID="f8ed9b991ab6667b1e03a69f211b723f7505f85df48c5eec4a6bcea80c431a52" Nov 01 15:00:16.492202 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:16.492176 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm_odh-ai-gateway-infra(e5269d12-6b1e-4153-b249-472a15a24d19)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" Nov 01 15:00:16.563695 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.563662 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428"] Nov 01 15:00:16.715543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:16.715353 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm"] Nov 01 15:00:17.075410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.075379 5473 scope.go:122] "RemoveContainer" containerID="a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce" Nov 01 15:00:17.075546 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.075477 5473 scope.go:122] "RemoveContainer" containerID="e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702" Nov 01 15:00:17.159117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.159090 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:17.320427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.320355 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph\" (UniqueName: \"kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph\") pod \"85a82f9f-afce-4221-80fe-f52cc264eee0\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " Nov 01 15:00:17.320789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.320548 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca\") pod \"85a82f9f-afce-4221-80fe-f52cc264eee0\" (UID: \"85a82f9f-afce-4221-80fe-f52cc264eee0\") " Nov 01 15:00:17.320899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.320867 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca" pod "85a82f9f-afce-4221-80fe-f52cc264eee0" (UID: "85a82f9f-afce-4221-80fe-f52cc264eee0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:00:17.321649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.321628 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a82f9f-afce-4221-80fe-f52cc264eee0-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:17.352711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.352676 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph" pod "85a82f9f-afce-4221-80fe-f52cc264eee0" (UID: "85a82f9f-afce-4221-80fe-f52cc264eee0"). InnerVolumeSpecName "kube-api-access-m6zph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:00:17.423096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.423063 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6zph\" (UniqueName: \"kubernetes.io/projected/85a82f9f-afce-4221-80fe-f52cc264eee0-kube-api-access-m6zph\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:17.562538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.562510 5473 generic.go:356] "Generic (PLEG): container finished" podID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerID="dfacec8f42affe053c543bb2cea4e90286cf6d6fc2cc771a16f79b8bfc718641" exitCode=60 Nov 01 15:00:17.562687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.562579 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" event={"ID":"3440bb1b-53b2-44ec-91b4-841dd3d7ec22","Type":"ContainerDied","Data":"dfacec8f42affe053c543bb2cea4e90286cf6d6fc2cc771a16f79b8bfc718641"} Nov 01 15:00:17.562687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.562640 5473 scope.go:122] "RemoveContainer" containerID="c6faaca863b4fb0bc209706c2afc71523224a21001caa4407ec1e951a3d5aec3" Nov 01 15:00:17.563069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.563051 5473 scope.go:122] "RemoveContainer" containerID="dfacec8f42affe053c543bb2cea4e90286cf6d6fc2cc771a16f79b8bfc718641" Nov 01 15:00:17.563413 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:17.563379 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww_odh-ai-gateway-infra(3440bb1b-53b2-44ec-91b4-841dd3d7ec22)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" Nov 01 15:00:17.564496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.564476 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" Nov 01 15:00:17.564665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.564558 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428" event={"ID":"85a82f9f-afce-4221-80fe-f52cc264eee0","Type":"ContainerDied","Data":"0602e89f404504533f6952ef2dcdc837a9b30657d216324423a9ac91f709ac4b"} Nov 01 15:00:17.576903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.576884 5473 scope.go:122] "RemoveContainer" containerID="ca97a479d037195a7236ed324bdbcd4a78aab642697b5b4f3387d914d69bf279" Nov 01 15:00:17.815964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.815939 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:17.835719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.835638 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm\" (UniqueName: \"kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm\") pod \"e5269d12-6b1e-4153-b249-472a15a24d19\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " Nov 01 15:00:17.835860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.835773 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca\") pod \"e5269d12-6b1e-4153-b249-472a15a24d19\" (UID: \"e5269d12-6b1e-4153-b249-472a15a24d19\") " Nov 01 15:00:17.836195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.836164 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca" pod "e5269d12-6b1e-4153-b249-472a15a24d19" (UID: "e5269d12-6b1e-4153-b249-472a15a24d19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:00:17.836392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.836372 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5269d12-6b1e-4153-b249-472a15a24d19-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:17.837812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.837784 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm" pod "e5269d12-6b1e-4153-b249-472a15a24d19" (UID: "e5269d12-6b1e-4153-b249-472a15a24d19"). InnerVolumeSpecName "kube-api-access-r9cdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:00:17.916678 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.916633 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428"] Nov 01 15:00:17.923607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.923579 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892420-t6428"] Nov 01 15:00:17.937511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:17.937476 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9cdm\" (UniqueName: \"kubernetes.io/projected/e5269d12-6b1e-4153-b249-472a15a24d19-kube-api-access-r9cdm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:18.075283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.074757 5473 scope.go:122] "RemoveContainer" containerID="c4394d941efc96a9508d7c697019d0a8d34dbc794ec58533723f85da37e4025b" Nov 01 15:00:18.582187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.582152 5473 generic.go:356] "Generic (PLEG): container finished" podID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerID="30fc3bba6e3be8aa18b5b9a3e6f17cbeccb26cc2999791fdcfadf2783a2a1945" exitCode=60 Nov 01 15:00:18.582690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.582302 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" event={"ID":"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f","Type":"ContainerDied","Data":"30fc3bba6e3be8aa18b5b9a3e6f17cbeccb26cc2999791fdcfadf2783a2a1945"} Nov 01 15:00:18.582690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.582360 5473 scope.go:122] "RemoveContainer" containerID="a1e71a731009cfffb7655f1b05607764411c72545dc573ef1c48ee946d8b86ce" Nov 01 15:00:18.582830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.582809 5473 scope.go:122] "RemoveContainer" containerID="30fc3bba6e3be8aa18b5b9a3e6f17cbeccb26cc2999791fdcfadf2783a2a1945" Nov 01 15:00:18.583228 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:18.583186 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww_odh-ai-gateway-infra(e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" Nov 01 15:00:18.586180 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.586149 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" event={"ID":"e5269d12-6b1e-4153-b249-472a15a24d19","Type":"ContainerDied","Data":"87b219551fb82c5f119780a79e52a9881efd34ac341a83458583b405fb4083aa"} Nov 01 15:00:18.586301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.586274 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm" Nov 01 15:00:18.595192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.595125 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerStarted","Data":"8eea919c3f7d093d687fc305424e4eb1995db403f9d1cf112291638f4187be50"} Nov 01 15:00:18.603786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.603696 5473 scope.go:122] "RemoveContainer" containerID="f8ed9b991ab6667b1e03a69f211b723f7505f85df48c5eec4a6bcea80c431a52" Nov 01 15:00:18.609797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.609705 5473 generic.go:356] "Generic (PLEG): container finished" podID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerID="6f6b952fef5a670fa2eb7a31139f5bb976d6819eaf602a006f673c09426ef9e8" exitCode=60 Nov 01 15:00:18.609899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.609831 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" event={"ID":"e998b9dd-027d-42bc-a948-93ed4ae6c0e8","Type":"ContainerDied","Data":"6f6b952fef5a670fa2eb7a31139f5bb976d6819eaf602a006f673c09426ef9e8"} Nov 01 15:00:18.610295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.610274 5473 scope.go:122] "RemoveContainer" containerID="6f6b952fef5a670fa2eb7a31139f5bb976d6819eaf602a006f673c09426ef9e8" Nov 01 15:00:18.610622 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:18.610598 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4_odh-ai-gateway-infra(e998b9dd-027d-42bc-a948-93ed4ae6c0e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" Nov 01 15:00:18.622376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.622356 5473 scope.go:122] "RemoveContainer" containerID="e6c89dfc7d9351bc99bb8f1127caa44cd1751c75f6849e9b30bb312dcb930702" Nov 01 15:00:18.698976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.698888 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww"] Nov 01 15:00:18.883615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.883586 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:18.959444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.959358 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds\" (UniqueName: \"kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds\") pod \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " Nov 01 15:00:18.959620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.959482 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca\") pod \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\" (UID: \"3440bb1b-53b2-44ec-91b4-841dd3d7ec22\") " Nov 01 15:00:18.960133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.960102 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca" pod "3440bb1b-53b2-44ec-91b4-841dd3d7ec22" (UID: "3440bb1b-53b2-44ec-91b4-841dd3d7ec22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:00:18.960595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.960575 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:18.962426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:18.962400 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds" pod "3440bb1b-53b2-44ec-91b4-841dd3d7ec22" (UID: "3440bb1b-53b2-44ec-91b4-841dd3d7ec22"). InnerVolumeSpecName "kube-api-access-5kvds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:00:19.064691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.064600 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvds\" (UniqueName: \"kubernetes.io/projected/3440bb1b-53b2-44ec-91b4-841dd3d7ec22-kube-api-access-5kvds\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:19.084137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.084101 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" path="/var/lib/kubelet/pods/85a82f9f-afce-4221-80fe-f52cc264eee0/volumes" Nov 01 15:00:19.097866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.097837 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm"] Nov 01 15:00:19.102593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.102565 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892420-xq9jm"] Nov 01 15:00:19.632282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.632245 5473 generic.go:356] "Generic (PLEG): container finished" podID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerID="8eea919c3f7d093d687fc305424e4eb1995db403f9d1cf112291638f4187be50" exitCode=60 Nov 01 15:00:19.632857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.632529 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerDied","Data":"8eea919c3f7d093d687fc305424e4eb1995db403f9d1cf112291638f4187be50"} Nov 01 15:00:19.632982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.632903 5473 scope.go:122] "RemoveContainer" containerID="c4394d941efc96a9508d7c697019d0a8d34dbc794ec58533723f85da37e4025b" Nov 01 15:00:19.633492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.633376 5473 scope.go:122] "RemoveContainer" containerID="8eea919c3f7d093d687fc305424e4eb1995db403f9d1cf112291638f4187be50" Nov 01 15:00:19.634029 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:19.633745 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2_odh-ai-gateway-infra(7dff3765-181b-45b4-9c5a-358c2c543d2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" Nov 01 15:00:19.637288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.637178 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" event={"ID":"3440bb1b-53b2-44ec-91b4-841dd3d7ec22","Type":"ContainerDied","Data":"610e0d79f82b80adbe71a3e72bf67a7fcbed07b5ff7ae4ea779edc54f70e8e08"} Nov 01 15:00:19.637288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.637202 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww" Nov 01 15:00:19.654436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.653479 5473 scope.go:122] "RemoveContainer" containerID="dfacec8f42affe053c543bb2cea4e90286cf6d6fc2cc771a16f79b8bfc718641" Nov 01 15:00:19.716448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.716401 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww"] Nov 01 15:00:19.877661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.877581 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww"] Nov 01 15:00:19.888766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.888616 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892420-vvdww"] Nov 01 15:00:19.895292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.891724 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4"] Nov 01 15:00:19.986102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.985690 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:19.997951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:19.997798 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2"] Nov 01 15:00:20.090883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.090854 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl\" (UniqueName: \"kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl\") pod \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " Nov 01 15:00:20.091064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.090927 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca\") pod \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\" (UID: \"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f\") " Nov 01 15:00:20.091459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.091407 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca" pod "e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" (UID: "e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:00:20.093357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.093327 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl" pod "e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" (UID: "e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f"). InnerVolumeSpecName "kube-api-access-tdmkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:00:20.095210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.095193 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:20.217292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.213304 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca\") pod \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " Nov 01 15:00:20.217292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.213545 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc\" (UniqueName: \"kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc\") pod \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\" (UID: \"e998b9dd-027d-42bc-a948-93ed4ae6c0e8\") " Nov 01 15:00:20.217292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.214964 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca" pod "e998b9dd-027d-42bc-a948-93ed4ae6c0e8" (UID: "e998b9dd-027d-42bc-a948-93ed4ae6c0e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:00:20.220050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.219952 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:20.220050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.219992 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmkl\" (UniqueName: \"kubernetes.io/projected/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-kube-api-access-tdmkl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:20.220050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.220021 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:20.260208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.260150 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc" pod "e998b9dd-027d-42bc-a948-93ed4ae6c0e8" (UID: "e998b9dd-027d-42bc-a948-93ed4ae6c0e8"). InnerVolumeSpecName "kube-api-access-mncfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:00:20.321828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.321792 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mncfc\" (UniqueName: \"kubernetes.io/projected/e998b9dd-027d-42bc-a948-93ed4ae6c0e8-kube-api-access-mncfc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:20.650762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.650725 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" Nov 01 15:00:20.650762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.650751 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4" event={"ID":"e998b9dd-027d-42bc-a948-93ed4ae6c0e8","Type":"ContainerDied","Data":"39e3c18e389083336488399c985239ec2aa5ac6fb27ac8bfcd95af1a19c0c3fa"} Nov 01 15:00:20.651357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.650825 5473 scope.go:122] "RemoveContainer" containerID="6f6b952fef5a670fa2eb7a31139f5bb976d6819eaf602a006f673c09426ef9e8" Nov 01 15:00:20.652779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.652750 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" event={"ID":"e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f","Type":"ContainerDied","Data":"6cc5b4033e646f3236b1463c9a926fd1ab001d0c9cf044148344fad18c6a2425"} Nov 01 15:00:20.652865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.652846 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww" Nov 01 15:00:20.666139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.666121 5473 scope.go:122] "RemoveContainer" containerID="30fc3bba6e3be8aa18b5b9a3e6f17cbeccb26cc2999791fdcfadf2783a2a1945" Nov 01 15:00:20.821499 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.821472 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4"] Nov 01 15:00:20.827126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.827050 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892420-v9jq4"] Nov 01 15:00:20.888398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.888364 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:20.920713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:20.920630 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:21.041932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.041898 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca\") pod \"7dff3765-181b-45b4-9c5a-358c2c543d2a\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " Nov 01 15:00:21.042138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.042115 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq\" (UniqueName: \"kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq\") pod \"7dff3765-181b-45b4-9c5a-358c2c543d2a\" (UID: \"7dff3765-181b-45b4-9c5a-358c2c543d2a\") " Nov 01 15:00:21.042431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.042398 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca" pod "7dff3765-181b-45b4-9c5a-358c2c543d2a" (UID: "7dff3765-181b-45b4-9c5a-358c2c543d2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:00:21.042701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.042679 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dff3765-181b-45b4-9c5a-358c2c543d2a-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:21.044308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.044283 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq" pod "7dff3765-181b-45b4-9c5a-358c2c543d2a" (UID: "7dff3765-181b-45b4-9c5a-358c2c543d2a"). InnerVolumeSpecName "kube-api-access-vzsfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:00:21.061579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.061552 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww"] Nov 01 15:00:21.098189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.098146 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" path="/var/lib/kubelet/pods/3440bb1b-53b2-44ec-91b4-841dd3d7ec22/volumes" Nov 01 15:00:21.098852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.098830 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" path="/var/lib/kubelet/pods/e5269d12-6b1e-4153-b249-472a15a24d19/volumes" Nov 01 15:00:21.099504 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.099485 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" path="/var/lib/kubelet/pods/e998b9dd-027d-42bc-a948-93ed4ae6c0e8/volumes" Nov 01 15:00:21.100621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.100600 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892420-gjtww"] Nov 01 15:00:21.145433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.145396 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzsfq\" (UniqueName: \"kubernetes.io/projected/7dff3765-181b-45b4-9c5a-358c2c543d2a-kube-api-access-vzsfq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:00:21.674718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.674686 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" Nov 01 15:00:21.674718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.674701 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2" event={"ID":"7dff3765-181b-45b4-9c5a-358c2c543d2a","Type":"ContainerDied","Data":"9b85ee14b21336ceeeb91c57b86dc110da02c431bd64bc0c148802548ab97653"} Nov 01 15:00:21.675281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.674766 5473 scope.go:122] "RemoveContainer" containerID="8eea919c3f7d093d687fc305424e4eb1995db403f9d1cf112291638f4187be50" Nov 01 15:00:21.840852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.840824 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2"] Nov 01 15:00:21.846648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:21.846625 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892420-rg9l2"] Nov 01 15:00:23.359861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.082983 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" path="/var/lib/kubelet/pods/7dff3765-181b-45b4-9c5a-358c2c543d2a/volumes" Nov 01 15:00:23.359861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.083438 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" path="/var/lib/kubelet/pods/e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f/volumes" Nov 01 15:00:23.696467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.696381 5473 generic.go:356] "Generic (PLEG): container finished" podID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerID="bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77" exitCode=0 Nov 01 15:00:23.696467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.696456 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" event={"ID":"80934e4d-60a9-410d-9e9d-8ac3320fa74d","Type":"ContainerDied","Data":"bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77"} Nov 01 15:00:23.696901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.696879 5473 scope.go:122] "RemoveContainer" containerID="bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77" Nov 01 15:00:23.698431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.698411 5473 generic.go:356] "Generic (PLEG): container finished" podID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerID="7199649c39a12ee9165c54518af6622d80b959797167204c761c903ec54df7a4" exitCode=0 Nov 01 15:00:23.698519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.698430 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" event={"ID":"093f8eb7-53cb-4245-9a53-888b2f8c8b4c","Type":"ContainerDied","Data":"7199649c39a12ee9165c54518af6622d80b959797167204c761c903ec54df7a4"} Nov 01 15:00:23.698519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.698409 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.698593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.698562 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.698636 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.698611 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.698789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.698773 5473 scope.go:122] "RemoveContainer" containerID="7199649c39a12ee9165c54518af6622d80b959797167204c761c903ec54df7a4" Nov 01 15:00:23.700201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.700173 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.700299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.700249 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.700299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.700283 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.700869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.700849 5473 generic.go:356] "Generic (PLEG): container finished" podID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerID="4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00" exitCode=0 Nov 01 15:00:23.700982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.700959 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" event={"ID":"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b","Type":"ContainerDied","Data":"4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00"} Nov 01 15:00:23.701414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.701395 5473 scope.go:122] "RemoveContainer" containerID="4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00" Nov 01 15:00:23.708779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.708749 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.708862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.708826 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:23.708905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:23.708873 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:24.711425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.711391 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" event={"ID":"093f8eb7-53cb-4245-9a53-888b2f8c8b4c","Type":"ContainerStarted","Data":"257955cceadfe33656c8fa77004b16439e8e1bca16d8d9e80c8dc1138e7dabf6"} Nov 01 15:00:24.713521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.713494 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" event={"ID":"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b","Type":"ContainerStarted","Data":"6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368"} Nov 01 15:00:24.715603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.715580 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" event={"ID":"80934e4d-60a9-410d-9e9d-8ac3320fa74d","Type":"ContainerStarted","Data":"b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320"} Nov 01 15:00:24.749241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.744907 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 15:00:24.749241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.746441 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 15:00:24.749241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.746507 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 15:00:24.898253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.882610 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 15:00:24.898253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.884319 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 15:00:24.898253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:24.884371 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 15:00:25.169238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.168662 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 15:00:25.169238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.168806 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 15:00:25.169238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.168875 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 15:00:25.677239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.676978 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:25.745241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.744556 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 15:00:25.745241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.744622 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 15:00:25.888241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.879545 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 15:00:25.888241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:25.879605 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 15:00:26.165239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.164742 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 15:00:26.165239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.164803 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 15:00:26.386243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.384165 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:26.536244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.535304 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 15:00:26.764239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.763569 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 15:00:26.764239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.763631 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 15:00:26.799304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.798850 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:26.885619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.885587 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 15:00:26.886003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:26.885876 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 15:00:27.161794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.161704 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 15:00:27.162080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.162047 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 15:00:27.279618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.279586 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp"] Nov 01 15:00:27.280734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.280711 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:27.280875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.280863 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:27.280972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.280960 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.281061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.281051 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.281143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.281133 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.292012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.291989 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.292193 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292179 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.292320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292307 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.292435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292423 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.292541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292530 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.292651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292641 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.292751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292740 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.292864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292854 5473 cpu_manager.go:414] "containerMap: removing container" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.292972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.292962 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.293072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293062 5473 cpu_manager.go:414] "containerMap: removing container" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.293169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293158 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.293283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293272 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:27.293396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293383 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:27.293593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293579 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.293707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293694 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.293821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293810 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.293927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.293915 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.294069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294058 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.294174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294163 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.294290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294279 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:27.294395 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294384 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:27.294502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294493 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:27.294603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294593 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:27.294906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.294893 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:27.295026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295016 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:27.295136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295125 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.295255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295244 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:27.295375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295364 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.295481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295470 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.295586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295576 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:27.295685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295675 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.295786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295776 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:27.295892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295881 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.295998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.295987 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.296107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.296097 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:27.296265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.296254 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:27.296388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.296378 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.296508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.296498 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3440bb1b-53b2-44ec-91b4-841dd3d7ec22" containerName="cleanup" Nov 01 15:00:27.311512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.311022 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.402240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.401584 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ca0f9e-openshift-default-dockercfg-7zqxz\"" Nov 01 15:00:27.402240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.402022 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.407961 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408020 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408077 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408125 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408177 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408323 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408367 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn9ws\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408396 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.418282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.408491 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.444240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.438768 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp"] Nov 01 15:00:27.562239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.557399 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613112 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613176 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn9ws\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613236 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613360 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613478 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613512 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613575 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613607 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.613666 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.643863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.642919 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.664979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.662370 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.664979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.664494 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.664979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.664725 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.664979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.664938 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.710762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.708826 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.731252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.720155 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn9ws\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.746818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.746783 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.758237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.750477 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy\") pod \"e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:27.778264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.771348 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 15:00:27.778264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.771409 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 15:00:27.783248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.781687 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:27.881241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.880605 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 15:00:27.881241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.880675 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 15:00:28.000294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:27.999447 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:28.176012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.159628 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 15:00:28.176012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.159686 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 15:00:28.601285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.586022 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp"] Nov 01 15:00:28.607568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.606542 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:28.607568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.606634 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:28.607568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.606679 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:28.746241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.745596 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 15:00:28.746241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.745653 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 15:00:28.812675 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.812648 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" event={"ID":"46ceda98-4397-4c78-91ed-655e5fe7523e","Type":"ContainerStarted","Data":"3867a83777a2a99fe44c4824d8ca77691428b1dfe49cfe2441ff590007fb2562"} Nov 01 15:00:28.905814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.905732 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 15:00:28.906084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:28.906055 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 15:00:29.161296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.160637 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 15:00:29.161296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.160709 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 15:00:29.744237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.743614 5473 patch_prober.go:29] interesting pod/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 01 15:00:29.744237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.743676 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 01 15:00:29.886239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.881836 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Nov 01 15:00:29.886239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.881924 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Nov 01 15:00:29.899245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:29.888836 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" event={"ID":"46ceda98-4397-4c78-91ed-655e5fe7523e","Type":"ContainerStarted","Data":"b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55"} Nov 01 15:00:30.001528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.001442 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:30.020559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.020531 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:30.069265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.066390 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb"] Nov 01 15:00:30.070170 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.070143 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1246786d-m5fwb"] Nov 01 15:00:30.098747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.098691 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podStartSLOduration=3.098676747 podStartE2EDuration="3.098676747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:30.060157952 +0000 UTC m=+2569.118309796" watchObservedRunningTime="2026-11-01 15:00:30.098676747 +0000 UTC m=+2569.156828596" Nov 01 15:00:30.157168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.157130 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 01 15:00:30.157401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.157248 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 01 15:00:30.617286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.617251 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:30.748358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.748324 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 15:00:30.749499 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.749481 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 15:00:30.801920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.801895 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 15:00:30.879376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.879306 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 15:00:30.880291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.880268 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 15:00:30.897956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.897926 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:30.898685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:30.898664 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:00:31.083330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:31.083302 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81b8cbc-02ef-41e6-8717-d036c01b7eed" path="/var/lib/kubelet/pods/d81b8cbc-02ef-41e6-8717-d036c01b7eed/volumes" Nov 01 15:00:31.123942 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:31.120089 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:00:31.162123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:31.162051 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 15:00:31.163201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:31.163179 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 15:00:32.732665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:32.732624 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:33.058260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:33.058198 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:33.248342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:33.248310 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:34.184082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:34.184045 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:34.636258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:34.636209 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:35.359062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:35.359019 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:35.544087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:35.544047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:35.857928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:35.857887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:36.030115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.030083 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:36.244280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244180 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp"] Nov 01 15:00:36.244774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244761 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:36.244822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244776 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:36.244822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244788 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:36.244822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244795 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:36.244822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244809 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:36.244822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244815 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff3765-181b-45b4-9c5a-358c2c543d2a" containerName="cleanup" Nov 01 15:00:36.244986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244828 5473 cpu_manager.go:414] "containerMap: removing container" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:36.244986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244833 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a82f9f-afce-4221-80fe-f52cc264eee0" containerName="cleanup" Nov 01 15:00:36.244986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244916 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e998b9dd-027d-42bc-a948-93ed4ae6c0e8" containerName="cleanup" Nov 01 15:00:36.244986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244925 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e617dc74-0ae2-4e1f-9eb5-6a9a0a02181f" containerName="cleanup" Nov 01 15:00:36.244986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.244934 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5269d12-6b1e-4153-b249-472a15a24d19" containerName="cleanup" Nov 01 15:00:36.249634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.249617 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.302503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.301297 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp"] Nov 01 15:00:36.347119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.347013 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ca0f9e\"" Nov 01 15:00:36.347556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.347518 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ca0f9e-dockercfg-7rsnk\"" Nov 01 15:00:36.358104 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.358030 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28ws\" (UniqueName: \"kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws\") pod \"payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.358550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.358443 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.413366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.413331 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt"] Nov 01 15:00:36.419377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.419354 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.460424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.460392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28ws\" (UniqueName: \"kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws\") pod \"payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.460594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.460482 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.461270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.461206 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.467937 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.467907 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt"] Nov 01 15:00:36.517757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.517728 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28ws\" (UniqueName: \"kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws\") pod \"payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.561871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.561840 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.562026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.561890 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpp8r\" (UniqueName: \"kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r\") pod \"payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.563544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.563525 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:36.663869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.663832 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.664050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.663919 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpp8r\" (UniqueName: \"kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r\") pod \"payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.664682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.664657 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.728911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.728883 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpp8r\" (UniqueName: \"kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r\") pod \"payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.740125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.740097 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:36.938041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.938006 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp"] Nov 01 15:00:36.976229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:36.976192 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" event={"ID":"d800b5c5-160b-4c46-96ae-4956d1777d09","Type":"ContainerStarted","Data":"d86b5dc95001aa054dbb6da1b619d72468381cb8ce55599a6616105724c93a9a"} Nov 01 15:00:37.107923 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:37.107886 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252062aa_197d_4f99_b9e1_2c656bfe26bb.slice/crio-259728db4b58c8e9f0e3e062650f977ec25c377942f82c02fec20f1f18313cae WatchSource:0}: Error finding container 259728db4b58c8e9f0e3e062650f977ec25c377942f82c02fec20f1f18313cae: Status 404 returned error can't find the container with id 259728db4b58c8e9f0e3e062650f977ec25c377942f82c02fec20f1f18313cae Nov 01 15:00:37.112425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:37.112401 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt"] Nov 01 15:00:37.985335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:37.985292 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" event={"ID":"252062aa-197d-4f99-b9e1-2c656bfe26bb","Type":"ContainerStarted","Data":"89953ae20769c9b9b3f8acea9cde5703730aca434acada17c6b9d395fa2e8e5a"} Nov 01 15:00:37.985793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:37.985354 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" event={"ID":"252062aa-197d-4f99-b9e1-2c656bfe26bb","Type":"ContainerStarted","Data":"259728db4b58c8e9f0e3e062650f977ec25c377942f82c02fec20f1f18313cae"} Nov 01 15:00:37.985793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:37.985381 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:37.987155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:37.987129 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" event={"ID":"d800b5c5-160b-4c46-96ae-4956d1777d09","Type":"ContainerStarted","Data":"e8bf75a76b4d195dec5b097facd39101a67762095ba08e4f40935ddc8fd5bff4"} Nov 01 15:00:37.987294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:37.987202 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:38.098990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:38.098898 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" podStartSLOduration=2.098881361 podStartE2EDuration="2.098881361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:38.098202948 +0000 UTC m=+2577.156354798" watchObservedRunningTime="2026-11-01 15:00:38.098881361 +0000 UTC m=+2577.157033214" Nov 01 15:00:38.216788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:38.216710 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" podStartSLOduration=2.216682496 podStartE2EDuration="2.216682496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:38.212433245 +0000 UTC m=+2577.270585136" watchObservedRunningTime="2026-11-01 15:00:38.216682496 +0000 UTC m=+2577.274834358" Nov 01 15:00:39.527235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:39.527168 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:39.708001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:39.707940 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:39.904261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:39.904125 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:40.627724 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:40.627674 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:41.051995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:41.051958 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p"] Nov 01 15:00:41.102006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:41.101974 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p"] Nov 01 15:00:41.139898 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:00:41.139864 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-eda9c242-4pd9p\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="26179a37-0712-40ec-8d08-a824aa6fb6a2" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda9c242-4pd9p" Nov 01 15:00:42.617529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:42.617497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:42.785954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:42.785916 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:42.968528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:42.968441 5473 scope.go:122] "RemoveContainer" containerID="013397bffedda06ace2c463f4408b1c948d4fd95d8d0798b8f02d67a8d89a3c0" Nov 01 15:00:42.979069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:42.979045 5473 scope.go:122] "RemoveContainer" containerID="2a3314c6544228f6c232c798f20aac52d87421a5596e27228e4780998978ec4d" Nov 01 15:00:43.146463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:43.146403 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26179a37-0712-40ec-8d08-a824aa6fb6a2" path="/var/lib/kubelet/pods/26179a37-0712-40ec-8d08-a824aa6fb6a2/volumes" Nov 01 15:00:45.117459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:45.117426 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:45.283277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:45.283242 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:45.634150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:45.634111 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:48.998336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:48.998303 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:00:48.998807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:48.998372 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:00:49.243865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:49.243832 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:49.583493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:49.583462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:50.051948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.051917 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt"] Nov 01 15:00:50.057999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.057981 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.118627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.118587 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b3f06c-openshift-default-dockercfg-rj5c6\"" Nov 01 15:00:50.182839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.182800 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:50.186400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.186373 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt"] Nov 01 15:00:50.233517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233473 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtnb\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.233734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233637 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.233734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233667 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.233734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233717 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.233920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233760 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.233920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233864 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.233920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233899 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.234096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.233992 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.234096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.234015 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.335997 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336034 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336080 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336109 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336150 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336180 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336327 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336361 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336435 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtnb\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336608 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336632 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336937 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336914 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.336994 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.336945 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.337040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.337023 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.338550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.338519 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.338672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.338655 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.393961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.393928 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.394111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.394040 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtnb\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb\") pod \"e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.407616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.407593 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:50.530723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.530662 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:50.702962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.702879 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:50.877965 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:50.877935 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c16441_be77_4a41_b90d_77b04df67172.slice/crio-353b1efcee94730495bb33a04b85d6c46d26d24b71f264b4d6aec467e6dd5889 WatchSource:0}: Error finding container 353b1efcee94730495bb33a04b85d6c46d26d24b71f264b4d6aec467e6dd5889: Status 404 returned error can't find the container with id 353b1efcee94730495bb33a04b85d6c46d26d24b71f264b4d6aec467e6dd5889 Nov 01 15:00:50.880642 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.880612 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:50.880745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.880675 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:50.880745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.880703 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:50.882358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:50.882325 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt"] Nov 01 15:00:51.138926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:51.138890 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" event={"ID":"60c16441-be77-4a41-b90d-77b04df67172","Type":"ContainerStarted","Data":"e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640"} Nov 01 15:00:51.139452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:51.138946 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" event={"ID":"60c16441-be77-4a41-b90d-77b04df67172","Type":"ContainerStarted","Data":"353b1efcee94730495bb33a04b85d6c46d26d24b71f264b4d6aec467e6dd5889"} Nov 01 15:00:51.257103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:51.257006 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podStartSLOduration=1.256991931 podStartE2EDuration="1.256991931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:51.255512443 +0000 UTC m=+2590.313664297" watchObservedRunningTime="2026-11-01 15:00:51.256991931 +0000 UTC m=+2590.315143788" Nov 01 15:00:51.408258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:51.408151 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:51.414680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:51.414652 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:52.146434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:52.146401 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:52.147265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:52.147248 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:00:54.012741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:54.012709 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:54.226562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:54.226525 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:55.270750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:55.270712 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:55.443972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:55.443931 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:56.050038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.050006 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8"] Nov 01 15:00:56.053904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.053884 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fc1f995-cbdq8"] Nov 01 15:00:56.304392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.304314 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:56.723305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.723204 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:56.752390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.752357 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47"] Nov 01 15:00:56.756975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.756958 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.910615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.910585 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-8d6ea2-openshift-default-dockercfg-r798d\"" Nov 01 15:00:56.914026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.913993 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.914581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.914553 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.914779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.914764 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.914936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.914923 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.915048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.915034 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d8lj\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.915165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.915152 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.915296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.915282 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.915408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.915395 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.915566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.915553 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:56.919423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.919394 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:56.923754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:56.923024 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47"] Nov 01 15:00:57.017260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017230 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.017483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017287 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.017483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017338 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.017483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017363 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d8lj\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.017483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.017483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017417 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.017483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.017443 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.018290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.018254 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.018410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.018320 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.018410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.018357 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.018410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.018388 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.019156 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.019081 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.019320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.019256 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.020154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.019692 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.036371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.036332 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.040119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.040090 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.075882 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.075848 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.076105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.076080 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d8lj\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj\") pod \"e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.082430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.082399 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3ddcea-4a38-42e3-9e73-34766b708211" path="/var/lib/kubelet/pods/8c3ddcea-4a38-42e3-9e73-34766b708211/volumes" Nov 01 15:00:57.187229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.187193 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:57.371894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.371805 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:57.795416 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:00:57.795387 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad6e2bc_bd38_44a2_9a41_312d985738ce.slice/crio-e49a34e3d2b7a19922c136ff10e93d8ea1471ef0f69a7f45e79fc74634861594 WatchSource:0}: Error finding container e49a34e3d2b7a19922c136ff10e93d8ea1471ef0f69a7f45e79fc74634861594: Status 404 returned error can't find the container with id e49a34e3d2b7a19922c136ff10e93d8ea1471ef0f69a7f45e79fc74634861594 Nov 01 15:00:57.798407 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.798367 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:57.798496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.798453 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:57.798532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.798497 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:00:57.800007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:57.799978 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47"] Nov 01 15:00:58.200461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.200376 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" event={"ID":"3ad6e2bc-bd38-44a2-9a41-312d985738ce","Type":"ContainerStarted","Data":"d781aa1b05bcf4c1d6b9d7c18e6a6c2bfa393d831b7269bdaa47e08d8a45f394"} Nov 01 15:00:58.200461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.200447 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" event={"ID":"3ad6e2bc-bd38-44a2-9a41-312d985738ce","Type":"ContainerStarted","Data":"e49a34e3d2b7a19922c136ff10e93d8ea1471ef0f69a7f45e79fc74634861594"} Nov 01 15:00:58.337903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.337868 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:58.372363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.372331 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:58.378470 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.378439 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:58.424028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.423968 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podStartSLOduration=2.423953299 podStartE2EDuration="2.423953299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:00:58.370080829 +0000 UTC m=+2597.428232679" watchObservedRunningTime="2026-11-01 15:00:58.423953299 +0000 UTC m=+2597.482105149" Nov 01 15:00:58.503656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:58.503571 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:00:59.208179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:59.208141 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:00:59.209129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:00:59.209110 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:01:00.341513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:00.341480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:01.118208 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:01:01.118173 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:01:01.807610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:01.807572 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:02.001648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:02.001616 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:02.168175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:02.168092 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:05.028487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:05.028456 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:05.228631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:05.228597 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:10.103472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:10.103432 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:14.506855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:14.506824 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:17.050108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:17.050076 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m"] Nov 01 15:01:17.100092 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:17.100066 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16523292-bl44m"] Nov 01 15:01:18.916566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:18.916534 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:19.082242 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:19.082193 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e18c4f-cb89-4944-979a-d06891fadce9" path="/var/lib/kubelet/pods/38e18c4f-cb89-4944-979a-d06891fadce9/volumes" Nov 01 15:01:23.324149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:23.323593 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:27.780184 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:27.780150 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:27.976188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:27.976154 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:28.142052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:28.141969 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:31.107301 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:01:31.107268 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:01:32.425046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:32.425011 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:33.049842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:33.049808 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw"] Nov 01 15:01:33.099694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:33.099660 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b0251-fplrw"] Nov 01 15:01:35.082256 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:35.082200 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1f2f6d-22af-4c28-99d4-59c1ffb2d655" path="/var/lib/kubelet/pods/bd1f2f6d-22af-4c28-99d4-59c1ffb2d655/volumes" Nov 01 15:01:36.768113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:36.768080 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:36.963317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:36.963281 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:37.137057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:37.136984 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:41.504961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:41.504918 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:43.052187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:43.052155 5473 scope.go:122] "RemoveContainer" containerID="1e3112b612a06493059c192e09e9257ffe70fbafd64d60e4ffe4f544771bfee6" Nov 01 15:01:43.062410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:43.062393 5473 scope.go:122] "RemoveContainer" containerID="546f583c792d50a37e197c72739ccaccff8d8b89c15793507e246cd30ecbb18e" Nov 01 15:01:43.071993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:43.071959 5473 scope.go:122] "RemoveContainer" containerID="8e406613191e659f8929f4a9a2de7687afe5ffaf044167853c30d70ff3390eb4" Nov 01 15:01:43.081616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:43.081597 5473 scope.go:122] "RemoveContainer" containerID="584b3a84b73cd60f47fa7e165cd81a2b4e17769ae7376328cc37582e679758d9" Nov 01 15:01:46.058404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:46.058367 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:50.433245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:50.425757 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:53.052246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:53.052195 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd"] Nov 01 15:01:53.099368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:53.099329 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a5af36c-mhgpd"] Nov 01 15:01:54.154110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.154079 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:54.689410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.689378 5473 generic.go:356] "Generic (PLEG): container finished" podID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerID="b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55" exitCode=0 Nov 01 15:01:54.689615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.689451 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" event={"ID":"46ceda98-4397-4c78-91ed-655e5fe7523e","Type":"ContainerDied","Data":"b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55"} Nov 01 15:01:54.689874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.689859 5473 scope.go:122] "RemoveContainer" containerID="b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55" Nov 01 15:01:54.691352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.691327 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:01:54.691423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.691404 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:01:54.691462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:54.691438 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:01:55.085377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:55.085347 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60feb8c-9858-4c60-9001-bd7dd503c589" path="/var/lib/kubelet/pods/f60feb8c-9858-4c60-9001-bd7dd503c589/volumes" Nov 01 15:01:55.698710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:55.698674 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" event={"ID":"46ceda98-4397-4c78-91ed-655e5fe7523e","Type":"ContainerStarted","Data":"bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624"} Nov 01 15:01:55.848807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:55.848765 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:56.000279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:56.000166 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:01:56.000440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:56.000372 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 15:01:56.000440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:56.000426 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 15:01:57.000849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:57.000814 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 15:01:57.001309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:57.000890 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 15:01:57.100398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:57.100362 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:01:58.000409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:58.000360 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 15:01:58.000600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:58.000427 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 15:01:59.000379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:59.000350 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 15:01:59.000780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:59.000400 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 15:01:59.445239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:01:59.442203 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:00.000358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:00.000321 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Nov 01 15:02:00.000549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:00.000370 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Nov 01 15:02:00.898480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:00.898448 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:02:01.005173 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:01.005143 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:02:01.107372 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:02:01.107340 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:02:01.781791 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:01.781764 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:02:03.835978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:03.835949 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:04.065396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:04.065361 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:04.223817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:04.223736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:06.541354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:06.541318 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:08.825077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:08.825020 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:09.584317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:09.584283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:12.048568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:12.048538 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht"] Nov 01 15:02:12.099457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:12.099419 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc6585d-b9hht"] Nov 01 15:02:13.084361 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:13.084325 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015329eb-fabc-4993-a9f1-62f31728cc4f" path="/var/lib/kubelet/pods/015329eb-fabc-4993-a9f1-62f31728cc4f/volumes" Nov 01 15:02:13.650611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:13.650571 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:14.055439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:14.055404 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk"] Nov 01 15:02:14.098822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:14.098787 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-363ecad6-7ntjk"] Nov 01 15:02:15.083618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:15.083582 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e89169b-766a-413b-a844-bafa803a4911" path="/var/lib/kubelet/pods/9e89169b-766a-413b-a844-bafa803a4911/volumes" Nov 01 15:02:18.130863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:18.130762 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:21.934147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:21.934117 5473 generic.go:356] "Generic (PLEG): container finished" podID="60c16441-be77-4a41-b90d-77b04df67172" containerID="e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640" exitCode=0 Nov 01 15:02:21.934586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:21.934192 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" event={"ID":"60c16441-be77-4a41-b90d-77b04df67172","Type":"ContainerDied","Data":"e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640"} Nov 01 15:02:21.934586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:21.934571 5473 scope.go:122] "RemoveContainer" containerID="e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640" Nov 01 15:02:21.936060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:21.936014 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:02:21.936166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:21.936100 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:02:21.936166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:21.936139 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:02:22.146624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:22.146590 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:02:22.565406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:22.565372 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:22.957254 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:22.957115 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" event={"ID":"60c16441-be77-4a41-b90d-77b04df67172","Type":"ContainerStarted","Data":"1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3"} Nov 01 15:02:23.408762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:23.408729 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:02:23.410160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:23.410136 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 15:02:23.410241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:23.410184 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 15:02:24.408276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:24.408242 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 15:02:24.408668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:24.408298 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 15:02:25.053550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:25.053513 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk"] Nov 01 15:02:25.102515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:25.102481 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fb20cd-trzdk"] Nov 01 15:02:25.408984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:25.408909 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 15:02:25.408984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:25.408958 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 15:02:26.098629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:26.098597 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g"] Nov 01 15:02:26.102167 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:26.102140 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c72192e-xcr9g"] Nov 01 15:02:26.408544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:26.408463 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 15:02:26.408544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:26.408517 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 15:02:27.060997 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:27.060960 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:27.089199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:27.089160 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf" path="/var/lib/kubelet/pods/0c69b0ea-1e4f-40a9-867c-1f29b3da6ebf/volumes" Nov 01 15:02:27.090041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:27.090015 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9e7705-837f-4db5-8155-3d68b578895e" path="/var/lib/kubelet/pods/4e9e7705-837f-4db5-8155-3d68b578895e/volumes" Nov 01 15:02:27.408826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:27.408754 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Nov 01 15:02:27.408826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:27.408820 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Nov 01 15:02:28.049015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:28.048986 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s"] Nov 01 15:02:28.098759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:28.098731 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424717e3-6mp8s"] Nov 01 15:02:28.412367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:28.412300 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:02:28.412768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:28.412752 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:02:28.413456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:28.413432 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:02:29.083123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:29.083089 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44bb504f-6d27-47a9-96fe-671da3f09f4b" path="/var/lib/kubelet/pods/44bb504f-6d27-47a9-96fe-671da3f09f4b/volumes" Nov 01 15:02:31.108592 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:02:31.108557 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:02:31.463431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:31.463341 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:35.937789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:35.933929 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:40.572420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:40.572351 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:42.053826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:42.053787 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk"] Nov 01 15:02:42.098595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:42.098560 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-204d04cb-5kkpk"] Nov 01 15:02:43.082775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:43.082745 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8044957f-49e0-4ffe-aa43-6133e6f16629" path="/var/lib/kubelet/pods/8044957f-49e0-4ffe-aa43-6133e6f16629/volumes" Nov 01 15:02:43.126717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:43.126686 5473 scope.go:122] "RemoveContainer" containerID="63d96ad6876e088e6a6ac530ac160fff3bfef87e744357cb80a496512085ce32" Nov 01 15:02:43.137140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:43.137116 5473 scope.go:122] "RemoveContainer" containerID="361d4a671b5be6320075edfa1e3e273f69eac521a20ff9925ebe0be41ccb6708" Nov 01 15:02:43.147055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:43.147030 5473 scope.go:122] "RemoveContainer" containerID="86614e1c9fe82341f4ec9c06788b7ef34ae41f9726f44c91fd5ae98168364983" Nov 01 15:02:43.994269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:43.994172 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:02:43.996072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:43.996053 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:02:45.207882 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:45.207847 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:45.250372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:45.250340 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:02:45.250539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:45.250400 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:02:45.386473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:45.386433 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:47.165112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:47.165076 5473 generic.go:356] "Generic (PLEG): container finished" podID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerID="d781aa1b05bcf4c1d6b9d7c18e6a6c2bfa393d831b7269bdaa47e08d8a45f394" exitCode=0 Nov 01 15:02:47.165615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:47.165146 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" event={"ID":"3ad6e2bc-bd38-44a2-9a41-312d985738ce","Type":"ContainerDied","Data":"d781aa1b05bcf4c1d6b9d7c18e6a6c2bfa393d831b7269bdaa47e08d8a45f394"} Nov 01 15:02:47.165615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:47.165599 5473 scope.go:122] "RemoveContainer" containerID="d781aa1b05bcf4c1d6b9d7c18e6a6c2bfa393d831b7269bdaa47e08d8a45f394" Nov 01 15:02:47.167315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:47.167286 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:02:47.167388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:47.167362 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:02:47.167433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:47.167393 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:02:48.174991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:48.174951 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" event={"ID":"3ad6e2bc-bd38-44a2-9a41-312d985738ce","Type":"ContainerStarted","Data":"9b51239a698e32d6696f65b99eea4c849019759c6ccd927d3d98c5f185b99473"} Nov 01 15:02:48.372052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:48.372011 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:02:48.372294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:48.372270 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 15:02:48.372348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:48.372323 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 15:02:49.373111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:49.373073 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 15:02:49.373538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:49.373156 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 15:02:49.944413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:49.942680 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:50.373023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:50.372939 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 15:02:50.373179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:50.373025 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 15:02:51.372162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:51.372121 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 15:02:51.372378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:51.372182 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 15:02:52.273532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:52.273498 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:52.372885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:52.372849 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 15:02:52.373154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:52.372908 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 15:02:53.373009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:53.372976 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 01 15:02:53.373442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:53.373031 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 01 15:02:54.377028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:54.376994 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:02:54.377538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:54.377428 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:02:54.378059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:54.378040 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:02:54.510751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:54.510721 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:57.366709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:57.366663 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:02:58.900682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:02:58.900638 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:01.109290 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:03:01.109249 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:03:03.386346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:03.386313 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:04.049521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:04.049485 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw"] Nov 01 15:03:04.098873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:04.098838 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10cd46e2-nk5cw"] Nov 01 15:03:05.082762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:05.082722 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5027c9ef-a0b0-49a8-af14-dcd1390e3e00" path="/var/lib/kubelet/pods/5027c9ef-a0b0-49a8-af14-dcd1390e3e00/volumes" Nov 01 15:03:05.481265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:05.481159 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:07.800900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:07.800861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:07.973770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:07.973732 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:09.920919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:09.920885 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:10.502871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:10.502833 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:11.563102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:11.563072 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:12.553723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:12.553683 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:17.007044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:17.006994 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:18.049248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:18.049199 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67"] Nov 01 15:03:18.099424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:18.099388 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8e885c-42s67"] Nov 01 15:03:19.054866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:19.054826 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:19.086617 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:19.086584 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e500828-6d5d-4b42-816d-41aba06545b7" path="/var/lib/kubelet/pods/7e500828-6d5d-4b42-816d-41aba06545b7/volumes" Nov 01 15:03:20.049692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:20.049657 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt"] Nov 01 15:03:20.100112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:20.100079 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe0c9c71-d47gt"] Nov 01 15:03:20.713781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:20.713745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:21.085309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:21.085279 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71e795f-59fc-4780-b874-3c6611cdd720" path="/var/lib/kubelet/pods/c71e795f-59fc-4780-b874-3c6611cdd720/volumes" Nov 01 15:03:21.662088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:21.662035 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:26.118481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:26.118445 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:26.302386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:26.302337 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:26.467250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:26.467140 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:27.043855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:27.043820 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:30.919846 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:30.919812 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:31.049809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:31.049774 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk"] Nov 01 15:03:31.099362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:31.099328 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk"] Nov 01 15:03:31.108814 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:03:31.108783 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:03:31.131012 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:03:31.130976 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-d6e09fec-rffrk\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="87bd3dd2-7523-40bd-98d1-6915aea72761" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6e09fec-rffrk" Nov 01 15:03:32.050442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:32.050413 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n"] Nov 01 15:03:32.099821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:32.099792 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ffb320-qwx4n"] Nov 01 15:03:32.625981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:32.625944 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:33.081837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:33.081800 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b17a283-095b-4f13-a69a-75c58c20b806" path="/var/lib/kubelet/pods/5b17a283-095b-4f13-a69a-75c58c20b806/volumes" Nov 01 15:03:33.082277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:33.082259 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87bd3dd2-7523-40bd-98d1-6915aea72761" path="/var/lib/kubelet/pods/87bd3dd2-7523-40bd-98d1-6915aea72761/volumes" Nov 01 15:03:35.301305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:35.301242 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:39.843019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:39.842988 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:42.049483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:42.049448 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2"] Nov 01 15:03:42.099076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:42.099048 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90a5cb9b-49tm2"] Nov 01 15:03:43.082190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.082156 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec308bb-25e5-4cd8-aea5-7bc1a6be0e24" path="/var/lib/kubelet/pods/fec308bb-25e5-4cd8-aea5-7bc1a6be0e24/volumes" Nov 01 15:03:43.189438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.189405 5473 scope.go:122] "RemoveContainer" containerID="f0b99e385abed0c471235d043c5cfeec25b5b4201fe2428d0531c4ef409ceec7" Nov 01 15:03:43.200346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.200320 5473 scope.go:122] "RemoveContainer" containerID="9c583febbcdc9b3ad0d6ef9fda3c68abcecce0cf6a41ada411598f13ac171f1c" Nov 01 15:03:43.214664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.214635 5473 scope.go:122] "RemoveContainer" containerID="b3ace4dbfe6c4314945b21f337078cd64a761959673e1a0f66ca0aebc58e81f3" Nov 01 15:03:43.224836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.224813 5473 scope.go:122] "RemoveContainer" containerID="c620eceaaccf492fe952b1d3d57eea8d908a1a0a754deaf6b60bdfe7221be22b" Nov 01 15:03:43.233608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.233590 5473 scope.go:122] "RemoveContainer" containerID="709745cb877db132a2f17bb0b6a88fd67e39a25a3d6096ce9605f4b6397209b2" Nov 01 15:03:43.242086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.242066 5473 scope.go:122] "RemoveContainer" containerID="6dce77dc5fb6363067bb935bafaca06358066e42dc4a413f4a2997f23be7cb36" Nov 01 15:03:43.250467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:43.250450 5473 scope.go:122] "RemoveContainer" containerID="9d8dfbf7534a4ef7a7a222c737565cad97550ef5e39521a1a364188005290831" Nov 01 15:03:44.048093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:44.048064 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2"] Nov 01 15:03:44.097654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:44.097624 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c6151f-9wrx2"] Nov 01 15:03:44.285800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:44.285772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:45.082522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:45.082490 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1358c224-4d1e-4220-a9b6-5c5f91436d10" path="/var/lib/kubelet/pods/1358c224-4d1e-4220-a9b6-5c5f91436d10/volumes" Nov 01 15:03:47.007376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:47.007327 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:48.023942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:48.023911 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:48.747714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:48.747681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:49.472414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:49.472381 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:49.678197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:49.678167 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:53.420088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:53.420041 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:53.697686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:53.697602 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:57.684149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:57.684111 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:57.854638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:57.854605 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:03:58.105483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:58.105454 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5"] Nov 01 15:03:58.105675 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:58.105630 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" containerID="cri-o://915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a" gracePeriod=30 Nov 01 15:03:58.155372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:58.155339 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp"] Nov 01 15:03:58.155590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:58.155552 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" containerID="cri-o://47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c" gracePeriod=30 Nov 01 15:03:59.048644 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:59.048606 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr"] Nov 01 15:03:59.051930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:59.051903 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e421592-9qzkr"] Nov 01 15:03:59.081866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:03:59.081842 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca7b9ae-a55a-4d7f-b73c-958824fce64b" path="/var/lib/kubelet/pods/8ca7b9ae-a55a-4d7f-b73c-958824fce64b/volumes" Nov 01 15:04:01.107061 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:01.107028 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:04:01.752200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:01.752168 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:03.457551 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.449723 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:03.465029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.464997 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 15:04:03.500199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.500175 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 15:04:03.552199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552172 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552207 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552274 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552312 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552442 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.552520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552471 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552512 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552594 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552603 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.552653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552645 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552722 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo\") pod \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\" (UID: \"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97\") " Nov 01 15:04:03.552767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552725 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.552767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552649 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:04:03.552943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.552774 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.553337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.553310 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.553483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.553341 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.553483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.553357 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.553483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.553371 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.553483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.553386 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.554578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.554556 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:03.554689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.554629 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.554759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.554718 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "kube-api-access-mv4lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:03.554817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.554802 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo" pod "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" (UID: "5c30dc70-cf04-4e83-a48e-be8ee8ae9f97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:04:03.654024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.653936 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654020 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654092 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654124 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654152 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654195 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.654300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654287 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654347 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.654596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654443 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.654596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654537 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.654596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654587 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654629 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.654819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.654757 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert\") pod \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\" (UID: \"ae0c442d-f399-48f1-8bd6-9c38e916fd39\") " Nov 01 15:04:03.655174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655129 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:04:03.655393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655371 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655398 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655412 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655426 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4lh\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-kube-api-access-mv4lh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655441 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655454 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655469 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655481 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.655810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.655496 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.656606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.656580 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:03.656741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.656721 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "kube-api-access-ffv98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:03.656848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.656829 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:04:03.657723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.657706 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy" pod "ae0c442d-f399-48f1-8bd6-9c38e916fd39" (UID: "ae0c442d-f399-48f1-8bd6-9c38e916fd39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:03.756912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.756868 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffv98\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-kube-api-access-ffv98\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.756912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.756905 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.757175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.756922 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.757175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.756938 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae0c442d-f399-48f1-8bd6-9c38e916fd39-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.759072 5473 generic.go:356] "Generic (PLEG): container finished" podID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerID="47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c" exitCode=0 Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.759251 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" event={"ID":"ae0c442d-f399-48f1-8bd6-9c38e916fd39","Type":"ContainerDied","Data":"47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c"} Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.759337 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" event={"ID":"ae0c442d-f399-48f1-8bd6-9c38e916fd39","Type":"ContainerDied","Data":"108038e8a019f412b407c8208b02d536fe5e1c4c12d34b5b45dbcaad15d8ddbd"} Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.759369 5473 scope.go:122] "RemoveContainer" containerID="47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c" Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.759520 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp" Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.763978 5473 generic.go:356] "Generic (PLEG): container finished" podID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerID="915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a" exitCode=0 Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.764021 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" event={"ID":"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97","Type":"ContainerDied","Data":"915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a"} Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.764044 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" Nov 01 15:04:03.767596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.764090 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5" event={"ID":"5c30dc70-cf04-4e83-a48e-be8ee8ae9f97","Type":"ContainerDied","Data":"87fbb7d4b72cf1c8f879fab809cc7dcfcb2d4686242aeb654712ce017f442bf8"} Nov 01 15:04:03.773942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.773923 5473 scope.go:122] "RemoveContainer" containerID="47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c" Nov 01 15:04:03.774208 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:03.774191 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c\": container with ID starting with 47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c not found: ID does not exist" containerID="47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c" Nov 01 15:04:03.774324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.774227 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c"} err="failed to get container status \"47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c\": rpc error: code = NotFound desc = could not find container \"47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c\": container with ID starting with 47b781515467c53ceb3290099f88af545fc697b5d910dd5f560ccba50ef0611c not found: ID does not exist" Nov 01 15:04:03.774324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.774244 5473 scope.go:122] "RemoveContainer" containerID="915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a" Nov 01 15:04:03.786478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.786453 5473 scope.go:122] "RemoveContainer" containerID="915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a" Nov 01 15:04:03.786766 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:03.786743 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a\": container with ID starting with 915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a not found: ID does not exist" containerID="915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a" Nov 01 15:04:03.786854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.786774 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a"} err="failed to get container status \"915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a\": rpc error: code = NotFound desc = could not find container \"915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a\": container with ID starting with 915ed64d9bff074524f73fe032b9e777fcf428cfea951daa05405e05df84ed5a not found: ID does not exist" Nov 01 15:04:03.930312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.930238 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp"] Nov 01 15:04:03.934040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:03.934009 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-4z2rp"] Nov 01 15:04:04.044015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:04.043973 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5"] Nov 01 15:04:04.047856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:04.047832 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-5n5z5"] Nov 01 15:04:05.073887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:05.073833 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:05.081954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:05.081928 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" path="/var/lib/kubelet/pods/5c30dc70-cf04-4e83-a48e-be8ee8ae9f97/volumes" Nov 01 15:04:05.082427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:05.082407 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" path="/var/lib/kubelet/pods/ae0c442d-f399-48f1-8bd6-9c38e916fd39/volumes" Nov 01 15:04:05.245317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:05.245287 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:06.528498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:06.528471 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:07.149747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:07.149703 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd"] Nov 01 15:04:07.149972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:07.149947 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" containerID="cri-o://6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368" gracePeriod=30 Nov 01 15:04:07.183766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:07.183735 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:07.915805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:07.915771 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:08.128888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:08.128859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:12.264103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.264066 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:12.433297 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.433269 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 15:04:12.467925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.467902 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.467959 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468005 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468043 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468108 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468148 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468187 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468301 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468394 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket\") pod \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\" (UID: \"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b\") " Nov 01 15:04:12.468599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468555 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:12.468599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468580 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:04:12.468708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468672 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:12.468768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468744 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:12.468850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.468820 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:12.469093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.469069 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.469235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.469095 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.469235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.469109 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.469235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.469122 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.469235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.469136 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.470323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.470298 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:12.470536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.470518 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "kube-api-access-shqjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:12.470619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.470601 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:04:12.470702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.470685 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token" pod "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" (UID: "c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:12.570844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.570814 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.570844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.570841 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqjc\" (UniqueName: \"kubernetes.io/projected/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-kube-api-access-shqjc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.570844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.570852 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.571071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.570861 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:12.832193 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.832159 5473 generic.go:356] "Generic (PLEG): container finished" podID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerID="6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368" exitCode=0 Nov 01 15:04:12.832380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.832235 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" Nov 01 15:04:12.832380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.832245 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" event={"ID":"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b","Type":"ContainerDied","Data":"6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368"} Nov 01 15:04:12.832380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.832334 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd" event={"ID":"c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b","Type":"ContainerDied","Data":"9b12f402efd0fb585ee8b0a3bb1e4c0a3224037c6a4ee21c88eae78ecd007850"} Nov 01 15:04:12.832380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.832361 5473 scope.go:122] "RemoveContainer" containerID="6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368" Nov 01 15:04:12.843705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.843686 5473 scope.go:122] "RemoveContainer" containerID="4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00" Nov 01 15:04:12.855796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.855775 5473 scope.go:122] "RemoveContainer" containerID="6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368" Nov 01 15:04:12.856116 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:12.856092 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368\": container with ID starting with 6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368 not found: ID does not exist" containerID="6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368" Nov 01 15:04:12.856180 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.856128 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368"} err="failed to get container status \"6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368\": rpc error: code = NotFound desc = could not find container \"6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368\": container with ID starting with 6ba5e8e4bb843db80a6c7f1b0cb4ae3cc2911484e1147a7c50145f0c0d433368 not found: ID does not exist" Nov 01 15:04:12.856180 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.856154 5473 scope.go:122] "RemoveContainer" containerID="4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00" Nov 01 15:04:12.856435 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:12.856417 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00\": container with ID starting with 4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00 not found: ID does not exist" containerID="4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00" Nov 01 15:04:12.856479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:12.856442 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00"} err="failed to get container status \"4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00\": rpc error: code = NotFound desc = could not find container \"4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00\": container with ID starting with 4d7b5a880fdf8dbf0f8377ed9d1177d8d8cf83873d5c2d248b84953e3e471e00 not found: ID does not exist" Nov 01 15:04:13.013237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:13.012378 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd"] Nov 01 15:04:13.062260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:13.061700 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0c9d0a-openshift-default-6c588b4cfc-vd4nd"] Nov 01 15:04:13.083741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:13.083655 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" path="/var/lib/kubelet/pods/c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b/volumes" Nov 01 15:04:13.328149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:13.328090 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:13.523907 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:13.523872 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:13.756728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:13.756694 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:14.243257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:14.224237 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:14.404522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:14.404486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:15.207313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:15.207283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:16.980007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:16.979967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:17.159047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:17.159004 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:18.393412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:18.393377 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:18.870625 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:18.870593 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:19.467326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.467296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:19.951091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.951061 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:19.959437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.959411 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6"] Nov 01 15:04:19.959976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.959963 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" Nov 01 15:04:19.960034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.959978 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" Nov 01 15:04:19.960034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.959999 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" Nov 01 15:04:19.960034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960005 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" Nov 01 15:04:19.960034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960028 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" Nov 01 15:04:19.960034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960033 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" Nov 01 15:04:19.960191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960053 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" Nov 01 15:04:19.960191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960058 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" Nov 01 15:04:19.960191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960138 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c30dc70-cf04-4e83-a48e-be8ee8ae9f97" containerName="istio-proxy" Nov 01 15:04:19.960191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960149 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0c442d-f399-48f1-8bd6-9c38e916fd39" containerName="istio-proxy" Nov 01 15:04:19.960191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960159 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" Nov 01 15:04:19.960191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.960165 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ca75c2-3332-4ab6-b5b5-a3f1c79ee72b" containerName="istio-proxy" Nov 01 15:04:19.964908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:19.964892 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.020900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.020868 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6"] Nov 01 15:04:20.066664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.066638 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7794bf-openshift-default-dockercfg-vw4r9\"" Nov 01 15:04:20.069276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069248 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mrh\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069307 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069349 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069430 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069476 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069552 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069610 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069660 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.069702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.069691 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.123274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.123248 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:20.171225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171196 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171265 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171339 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mrh\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171371 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171397 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171423 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171466 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171504 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171537 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.171938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.171904 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.172065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.172022 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.172152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.172059 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.172152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.172075 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.172152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.172063 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.173641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.173619 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.173741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.173705 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.228545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.228487 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mrh\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.228647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.228618 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token\") pod \"e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.286237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.282467 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:20.303760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.298986 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:20.760440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.760406 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6"] Nov 01 15:04:20.785578 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:04:20.785553 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04fbb0_db09_4a7e_a4e4_19919e76b584.slice/crio-f2b0deb50af89ad2b9943a45047c7b99de2d336eca6d4e63a93958d1dd9aa80f WatchSource:0}: Error finding container f2b0deb50af89ad2b9943a45047c7b99de2d336eca6d4e63a93958d1dd9aa80f: Status 404 returned error can't find the container with id f2b0deb50af89ad2b9943a45047c7b99de2d336eca6d4e63a93958d1dd9aa80f Nov 01 15:04:20.788072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.788037 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:04:20.788152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.788106 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:04:20.788152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.788133 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:04:20.909766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.909735 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" event={"ID":"3a04fbb0-db09-4a7e-a4e4-19919e76b584","Type":"ContainerStarted","Data":"b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971"} Nov 01 15:04:20.909927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:20.909791 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" event={"ID":"3a04fbb0-db09-4a7e-a4e4-19919e76b584","Type":"ContainerStarted","Data":"f2b0deb50af89ad2b9943a45047c7b99de2d336eca6d4e63a93958d1dd9aa80f"} Nov 01 15:04:21.035846 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.035785 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podStartSLOduration=2.035765645 podStartE2EDuration="2.035765645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:21.030310551 +0000 UTC m=+2800.088462401" watchObservedRunningTime="2026-11-01 15:04:21.035765645 +0000 UTC m=+2800.093917497" Nov 01 15:04:21.283732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.283690 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:21.289441 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.289372 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:21.408461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.408433 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:21.811655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.811620 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47"] Nov 01 15:04:21.812130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.811860 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" containerID="cri-o://9b51239a698e32d6696f65b99eea4c849019759c6ccd927d3d98c5f185b99473" gracePeriod=30 Nov 01 15:04:21.859314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.859282 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:21.986669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.986639 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:21.991448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:21.991423 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:04:22.627694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:22.627662 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:23.084073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.084044 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs"] Nov 01 15:04:23.088197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.088177 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.140959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.140919 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1ac11c-openshift-default-dockercfg-zsr8q\"" Nov 01 15:04:23.158894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.158868 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs"] Nov 01 15:04:23.166637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.166585 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:23.257208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.257168 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.257536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.257511 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.257714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.257623 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.257714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.257664 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.261003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.259310 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.261003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.259462 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgr7\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.261003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.259527 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.261003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.259738 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.261003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.259798 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.360710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360594 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.360710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360652 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.360710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360687 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360713 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360769 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360794 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360824 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360903 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgr7\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.360933 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.361138 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.361190 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.361520 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.361613 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.361690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.361610 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.365237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.363462 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.365237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.363772 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.397229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.397187 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:23.418531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.418509 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.418717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.418697 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgr7\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7\") pod \"e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.707031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.706952 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:23.790145 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.790116 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:23.964667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:23.964587 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:24.122558 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:04:24.122529 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b62e39_2b10_41d3_80e5_a449d34d0f23.slice/crio-d2fb5673d4ef4a90af68a54cc68cc046710b7977dcd4a5cb48d0c576520bd6f1 WatchSource:0}: Error finding container d2fb5673d4ef4a90af68a54cc68cc046710b7977dcd4a5cb48d0c576520bd6f1: Status 404 returned error can't find the container with id d2fb5673d4ef4a90af68a54cc68cc046710b7977dcd4a5cb48d0c576520bd6f1 Nov 01 15:04:24.125434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.125402 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:04:24.125538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.125468 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:04:24.125538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.125498 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:04:24.125982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.125946 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs"] Nov 01 15:04:24.144145 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.143867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:24.380342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.380297 5473 patch_prober.go:29] interesting pod/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:04:24.380513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.380367 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:04:24.841517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:24.841483 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:25.008918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:25.008880 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" event={"ID":"43b62e39-2b10-41d3-80e5-a449d34d0f23","Type":"ContainerStarted","Data":"699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df"} Nov 01 15:04:25.009069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:25.008938 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" event={"ID":"43b62e39-2b10-41d3-80e5-a449d34d0f23","Type":"ContainerStarted","Data":"d2fb5673d4ef4a90af68a54cc68cc046710b7977dcd4a5cb48d0c576520bd6f1"} Nov 01 15:04:25.228325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:25.228211 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podStartSLOduration=2.228195315 podStartE2EDuration="2.228195315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:25.176232656 +0000 UTC m=+2804.234384497" watchObservedRunningTime="2026-11-01 15:04:25.228195315 +0000 UTC m=+2804.286347165" Nov 01 15:04:25.712757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:25.712712 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:25.718712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:25.718684 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:25.733148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:25.733116 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:26.020462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:26.020433 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:26.021474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:26.021451 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:04:27.031199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.031164 5473 generic.go:356] "Generic (PLEG): container finished" podID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerID="9b51239a698e32d6696f65b99eea4c849019759c6ccd927d3d98c5f185b99473" exitCode=0 Nov 01 15:04:27.031732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.031267 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" event={"ID":"3ad6e2bc-bd38-44a2-9a41-312d985738ce","Type":"ContainerDied","Data":"9b51239a698e32d6696f65b99eea4c849019759c6ccd927d3d98c5f185b99473"} Nov 01 15:04:27.031732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.031348 5473 scope.go:122] "RemoveContainer" containerID="d781aa1b05bcf4c1d6b9d7c18e6a6c2bfa393d831b7269bdaa47e08d8a45f394" Nov 01 15:04:27.079539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.079480 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:04:27.114620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114593 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114638 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114719 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114779 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114802 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114855 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114879 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:27.114993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114916 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.114993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.114989 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.115348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115024 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo\") pod \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\" (UID: \"3ad6e2bc-bd38-44a2-9a41-312d985738ce\") " Nov 01 15:04:27.115348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115135 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:27.115348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115257 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:27.115348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115170 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:04:27.115348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115332 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:27.115811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115771 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.115811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115795 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.115811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115811 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.116156 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115827 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.116156 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.115840 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.117349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.117325 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:27.117449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.117376 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:27.117628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.117612 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:04:27.117672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.117636 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj" pod "3ad6e2bc-bd38-44a2-9a41-312d985738ce" (UID: "3ad6e2bc-bd38-44a2-9a41-312d985738ce"). InnerVolumeSpecName "kube-api-access-6d8lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:27.217654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.217612 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d8lj\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-kube-api-access-6d8lj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.217654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.217655 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.217939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.217669 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:27.217939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:27.217683 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad6e2bc-bd38-44a2-9a41-312d985738ce-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:28.039694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:28.039664 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" Nov 01 15:04:28.040125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:28.039703 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47" event={"ID":"3ad6e2bc-bd38-44a2-9a41-312d985738ce","Type":"ContainerDied","Data":"e49a34e3d2b7a19922c136ff10e93d8ea1471ef0f69a7f45e79fc74634861594"} Nov 01 15:04:28.040125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:28.039750 5473 scope.go:122] "RemoveContainer" containerID="9b51239a698e32d6696f65b99eea4c849019759c6ccd927d3d98c5f185b99473" Nov 01 15:04:28.255210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:28.255179 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47"] Nov 01 15:04:28.305258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:28.305176 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-8d6ea2-openshift-default-67898cbf65-d2g47"] Nov 01 15:04:29.078078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.078048 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:29.081905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.081877 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" path="/var/lib/kubelet/pods/3ad6e2bc-bd38-44a2-9a41-312d985738ce/volumes" Nov 01 15:04:29.238559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.238525 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:29.445344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.445255 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:29.623271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.623190 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q"] Nov 01 15:04:29.626466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.626432 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" Nov 01 15:04:29.626466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.626465 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" Nov 01 15:04:29.626700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.626517 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" Nov 01 15:04:29.626700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.626526 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" Nov 01 15:04:29.626817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.626719 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" Nov 01 15:04:29.626817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.626742 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ad6e2bc-bd38-44a2-9a41-312d985738ce" containerName="istio-proxy" Nov 01 15:04:29.639177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.639151 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.641687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.641664 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:29.686311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.686266 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q"] Nov 01 15:04:29.743919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.729075 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4kq\" (UniqueName: \"kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq\") pod \"payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.744257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.744202 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.744395 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.743418 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7794bf\"" Nov 01 15:04:29.744438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.743376 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7794bf-dockercfg-nfbgq\"" Nov 01 15:04:29.818705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.818676 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:29.845783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.845756 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.845956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.845934 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4kq\" (UniqueName: \"kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq\") pod \"payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.846540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.846500 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.903568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.903539 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4kq\" (UniqueName: \"kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq\") pod \"payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:29.954357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:29.954330 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:30.330501 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:04:30.330471 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bacb08_bc51_462e_8b71_8f35b201a6a5.slice/crio-e35a56a0b79ad742f6691fbfb33a89737e2292d489386c403e41ebb17f8bb84c WatchSource:0}: Error finding container e35a56a0b79ad742f6691fbfb33a89737e2292d489386c403e41ebb17f8bb84c: Status 404 returned error can't find the container with id e35a56a0b79ad742f6691fbfb33a89737e2292d489386c403e41ebb17f8bb84c Nov 01 15:04:30.333928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:30.333901 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q"] Nov 01 15:04:31.081671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:31.081626 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" event={"ID":"04bacb08-bc51-462e-8b71-8f35b201a6a5","Type":"ContainerStarted","Data":"ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3"} Nov 01 15:04:31.081891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:31.081682 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" event={"ID":"04bacb08-bc51-462e-8b71-8f35b201a6a5","Type":"ContainerStarted","Data":"e35a56a0b79ad742f6691fbfb33a89737e2292d489386c403e41ebb17f8bb84c"} Nov 01 15:04:31.081891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:31.081712 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:31.108408 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:31.108377 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:04:31.194201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:31.194155 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" podStartSLOduration=2.1941416240000002 podStartE2EDuration="2.194141624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:04:31.193282582 +0000 UTC m=+2810.251434432" watchObservedRunningTime="2026-11-01 15:04:31.194141624 +0000 UTC m=+2810.252293500" Nov 01 15:04:34.231601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:34.231568 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:35.846709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:35.846672 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:36.007108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:36.007075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:38.607678 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:38.607647 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:42.999485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:42.999441 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:43.098377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.098351 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:04:43.306338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.306312 5473 scope.go:122] "RemoveContainer" containerID="7ad60ff30a4d79a219b2d5679a344383f0f74f4935ec25269fd32c919114470b" Nov 01 15:04:43.325313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.325287 5473 scope.go:122] "RemoveContainer" containerID="c6da71600de5ce3e5431d655bf5f42f7c45085a489f9df4e86141bf851717350" Nov 01 15:04:43.335127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.335102 5473 scope.go:122] "RemoveContainer" containerID="a0554e2cfbb3334a85faa9cdb4b2123959d20f82d5435d42923179ba6c535622" Nov 01 15:04:43.344379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.344361 5473 scope.go:122] "RemoveContainer" containerID="e0398d3822bed6c520238116bd40976cbbbb695018f976fe48e2d4714bcd831d" Nov 01 15:04:43.352939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.352919 5473 scope.go:122] "RemoveContainer" containerID="919e21d34687b99e2199cad6b389cbfcf4ca4e878dfa2b5dc0e4c7dfe2defa8a" Nov 01 15:04:43.361544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.361526 5473 scope.go:122] "RemoveContainer" containerID="4a503321d71f3886e948e7f04a6c5b8975a6797d8a532b2ba6b7a40a8ff6ae1c" Nov 01 15:04:43.369814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:43.369791 5473 scope.go:122] "RemoveContainer" containerID="63279d001b1581679d32f5108f35ab12e6420b00717a2524f24d25bd1abc5289" Nov 01 15:04:47.363769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:47.363736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:51.828670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:51.828632 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:52.054688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:52.054656 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk"] Nov 01 15:04:52.104745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:52.104674 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caebccb3-pcqhk"] Nov 01 15:04:53.082887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:53.082855 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c26cdb-a883-4683-bcf6-3d268794fd49" path="/var/lib/kubelet/pods/39c26cdb-a883-4683-bcf6-3d268794fd49/volumes" Nov 01 15:04:54.812923 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.812884 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22"] Nov 01 15:04:54.813431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.813077 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" containerID="cri-o://8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b" gracePeriod=30 Nov 01 15:04:54.863262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.863205 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl"] Nov 01 15:04:54.863457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.863427 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" containerID="cri-o://daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c" gracePeriod=30 Nov 01 15:04:54.866520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.866496 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7"] Nov 01 15:04:54.866717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.866696 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" containerID="cri-o://0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089" gracePeriod=30 Nov 01 15:04:54.913334 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.913300 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m"] Nov 01 15:04:54.913553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.913530 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" containerID="cri-o://4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4" gracePeriod=30 Nov 01 15:04:54.916447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.916422 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf"] Nov 01 15:04:54.916638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:54.916611 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" podUID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" containerName="discovery" containerID="cri-o://5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3" gracePeriod=30 Nov 01 15:04:55.191174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.191143 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 15:04:55.259598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.259563 5473 generic.go:356] "Generic (PLEG): container finished" podID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" containerID="5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3" exitCode=0 Nov 01 15:04:55.259804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.259611 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" Nov 01 15:04:55.259804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.259627 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" event={"ID":"f98b3fa4-5995-498c-9190-e87eb66aeb4a","Type":"ContainerDied","Data":"5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3"} Nov 01 15:04:55.259804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.259676 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf" event={"ID":"f98b3fa4-5995-498c-9190-e87eb66aeb4a","Type":"ContainerDied","Data":"323558dd5ed32cd67457fb0c287b87d70e50ee09e8fde16c7f2ff6bf9f6f7c53"} Nov 01 15:04:55.259804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.259703 5473 scope.go:122] "RemoveContainer" containerID="5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3" Nov 01 15:04:55.270006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.269984 5473 scope.go:122] "RemoveContainer" containerID="5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3" Nov 01 15:04:55.270307 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:04:55.270285 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3\": container with ID starting with 5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3 not found: ID does not exist" containerID="5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3" Nov 01 15:04:55.270368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.270317 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3"} err="failed to get container status \"5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3\": rpc error: code = NotFound desc = could not find container \"5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3\": container with ID starting with 5b17b9a2382c5d9ce33938c0ca0da7da988a20dd97d44156db9a0467012cc2a3 not found: ID does not exist" Nov 01 15:04:55.285905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.285877 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.285996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.285932 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.285996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.285966 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.286114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.285997 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.286114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.286053 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.286114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.286077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.286340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.286118 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts\") pod \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\" (UID: \"f98b3fa4-5995-498c-9190-e87eb66aeb4a\") " Nov 01 15:04:55.286529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.286498 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:04:55.286724 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.286696 5473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-ca-configmap\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.288434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.288408 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:04:55.288540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.288451 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:04:55.288667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.288645 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "kube-api-access-4hswz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:55.288756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.288663 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:04:55.288756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.288702 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:04:55.288873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.288861 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token" pod "f98b3fa4-5995-498c-9190-e87eb66aeb4a" (UID: "f98b3fa4-5995-498c-9190-e87eb66aeb4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:04:55.388332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.388203 5473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-csr-dns-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.388332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.388269 5473 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f98b3fa4-5995-498c-9190-e87eb66aeb4a-local-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.388332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.388282 5473 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-cacerts\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.388332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.388291 5473 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-kubeconfig\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.388332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.388299 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.388332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.388308 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hswz\" (UniqueName: \"kubernetes.io/projected/f98b3fa4-5995-498c-9190-e87eb66aeb4a-kube-api-access-4hswz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:04:55.685473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.685393 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf"] Nov 01 15:04:55.688927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:55.688900 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rq7xf"] Nov 01 15:04:56.213023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:56.212986 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:57.082529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:57.082493 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" path="/var/lib/kubelet/pods/f98b3fa4-5995-498c-9190-e87eb66aeb4a/volumes" Nov 01 15:04:57.998329 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:57.998281 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:04:58.561060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:58.561025 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:04:58.561252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:58.561084 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:04:59.147796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:59.147757 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:04:59.148264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:59.147815 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:04:59.947795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:59.947764 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 01 15:04:59.948049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:04:59.947835 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 01 15:05:00.236976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.236947 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 15:05:00.240419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.240398 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 15:05:00.253051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.253027 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 15:05:00.260828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.260808 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 15:05:00.307922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.307889 5473 generic.go:356] "Generic (PLEG): container finished" podID="fb918a81-0960-4265-baf2-285161272836" containerID="daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c" exitCode=0 Nov 01 15:05:00.308161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.307939 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" Nov 01 15:05:00.308161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.307973 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" event={"ID":"fb918a81-0960-4265-baf2-285161272836","Type":"ContainerDied","Data":"daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c"} Nov 01 15:05:00.308161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.308029 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" event={"ID":"fb918a81-0960-4265-baf2-285161272836","Type":"ContainerDied","Data":"7edf8342b8a96be51962e8e6263263e5b73dbdaddcba5917b9d253958002bf40"} Nov 01 15:05:00.308161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.308059 5473 scope.go:122] "RemoveContainer" containerID="daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c" Nov 01 15:05:00.309586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.309565 5473 generic.go:356] "Generic (PLEG): container finished" podID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerID="4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4" exitCode=0 Nov 01 15:05:00.309711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.309593 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" event={"ID":"ef256c70-9de7-4d17-b204-6239b96e14b3","Type":"ContainerDied","Data":"4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4"} Nov 01 15:05:00.309711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.309625 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" Nov 01 15:05:00.309711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.309637 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m" event={"ID":"ef256c70-9de7-4d17-b204-6239b96e14b3","Type":"ContainerDied","Data":"dcff0908c87f4ee95996107804ebd01a7f9f4e2a599b1ab1e71a26187780e5f8"} Nov 01 15:05:00.311413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.311385 5473 generic.go:356] "Generic (PLEG): container finished" podID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerID="8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b" exitCode=0 Nov 01 15:05:00.311510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.311413 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" event={"ID":"90ab68bc-e2ea-4021-974b-86ec3260e5a7","Type":"ContainerDied","Data":"8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b"} Nov 01 15:05:00.311510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.311430 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" Nov 01 15:05:00.311510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.311456 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22" event={"ID":"90ab68bc-e2ea-4021-974b-86ec3260e5a7","Type":"ContainerDied","Data":"6838e8382c39b2c942d75055f59a69378a67f1fdee4a08a6ccf8b3f40088dedd"} Nov 01 15:05:00.312981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.312963 5473 generic.go:356] "Generic (PLEG): container finished" podID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerID="0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089" exitCode=0 Nov 01 15:05:00.313073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.312999 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" Nov 01 15:05:00.313073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.312996 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" event={"ID":"641a2393-367a-44ab-a86f-7b44316c0d9c","Type":"ContainerDied","Data":"0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089"} Nov 01 15:05:00.313073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.313037 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7" event={"ID":"641a2393-367a-44ab-a86f-7b44316c0d9c","Type":"ContainerDied","Data":"c605186069988f7943ab9adc976f34652f6b1b361c24db8e66ae3773348c663b"} Nov 01 15:05:00.318715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.318697 5473 scope.go:122] "RemoveContainer" containerID="daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c" Nov 01 15:05:00.318968 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:00.318951 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c\": container with ID starting with daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c not found: ID does not exist" containerID="daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c" Nov 01 15:05:00.319009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.318976 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c"} err="failed to get container status \"daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c\": rpc error: code = NotFound desc = could not find container \"daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c\": container with ID starting with daade048133d7e94829c5e73a81977a5888f5c94e808337b96ed6019d997d37c not found: ID does not exist" Nov 01 15:05:00.319009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.318992 5473 scope.go:122] "RemoveContainer" containerID="4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4" Nov 01 15:05:00.331249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.331232 5473 scope.go:122] "RemoveContainer" containerID="4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4" Nov 01 15:05:00.331569 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:00.331547 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4\": container with ID starting with 4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4 not found: ID does not exist" containerID="4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4" Nov 01 15:05:00.331661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.331573 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4"} err="failed to get container status \"4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4\": rpc error: code = NotFound desc = could not find container \"4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4\": container with ID starting with 4bc6c7a7a86ec0af0c75345673c025465eb428edd6eba0fbd110af3f1e4b3bd4 not found: ID does not exist" Nov 01 15:05:00.331661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.331592 5473 scope.go:122] "RemoveContainer" containerID="8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b" Nov 01 15:05:00.341194 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.341172 5473 scope.go:122] "RemoveContainer" containerID="8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b" Nov 01 15:05:00.341462 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:00.341444 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b\": container with ID starting with 8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b not found: ID does not exist" containerID="8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b" Nov 01 15:05:00.341516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.341468 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b"} err="failed to get container status \"8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b\": rpc error: code = NotFound desc = could not find container \"8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b\": container with ID starting with 8cc09d60869a7adae452e7c568f80ac921780cefc5f7a2265dd8cf8c8c2bdf1b not found: ID does not exist" Nov 01 15:05:00.341516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.341485 5473 scope.go:122] "RemoveContainer" containerID="0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089" Nov 01 15:05:00.346236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346190 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.346332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346258 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.346332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346278 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.346332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346327 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.346515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346368 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.346515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346399 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.346515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346435 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.346685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346449 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.346685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346547 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.346685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346568 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.346685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346601 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.346685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346649 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.346685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346677 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346723 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346756 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346773 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346806 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346831 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346860 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346892 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346924 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.346975 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347018 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347080 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347119 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347150 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347205 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347249 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert\") pod \"fb918a81-0960-4265-baf2-285161272836\" (UID: \"fb918a81-0960-4265-baf2-285161272836\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347284 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347323 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347351 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347372 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347409 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347445 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347477 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347530 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347542 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347563 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347577 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347596 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347615 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo\") pod \"ef256c70-9de7-4d17-b204-6239b96e14b3\" (UID: \"ef256c70-9de7-4d17-b204-6239b96e14b3\") " Nov 01 15:05:00.347643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347648 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo\") pod \"641a2393-367a-44ab-a86f-7b44316c0d9c\" (UID: \"641a2393-367a-44ab-a86f-7b44316c0d9c\") " Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347699 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert\") pod \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\" (UID: \"90ab68bc-e2ea-4021-974b-86ec3260e5a7\") " Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347764 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.347804 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348004 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348211 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348484 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348544 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef256c70-9de7-4d17-b204-6239b96e14b3-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348567 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348582 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348597 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348612 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348626 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348640 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/641a2393-367a-44ab-a86f-7b44316c0d9c-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348654 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348668 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348672 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348682 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.348702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348698 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.349729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.348871 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.349729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.349462 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:05:00.349729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.349580 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "kube-api-access-fc886". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.349900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.349823 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.349900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.349843 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.350405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.350173 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:05:00.350802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.350775 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.351004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.350983 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.351088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.351069 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.351705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.351673 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.351705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.351693 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "kube-api-access-rbxlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.352976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.352923 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.353853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.353780 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:05:00.353853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.353787 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.353853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.353809 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:05:00.353853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.353849 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.355046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.354944 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:05:00.355046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.354963 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "kube-api-access-9fgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.355046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.354987 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "kube-api-access-h7c6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.355046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.354988 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy" pod "641a2393-367a-44ab-a86f-7b44316c0d9c" (UID: "641a2393-367a-44ab-a86f-7b44316c0d9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.355046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.355028 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token" pod "ef256c70-9de7-4d17-b204-6239b96e14b3" (UID: "ef256c70-9de7-4d17-b204-6239b96e14b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:05:00.355371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.355068 5473 scope.go:122] "RemoveContainer" containerID="0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089" Nov 01 15:05:00.355371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.355156 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo" pod "90ab68bc-e2ea-4021-974b-86ec3260e5a7" (UID: "90ab68bc-e2ea-4021-974b-86ec3260e5a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:05:00.355371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.355275 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy" pod "fb918a81-0960-4265-baf2-285161272836" (UID: "fb918a81-0960-4265-baf2-285161272836"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:05:00.355490 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:00.355459 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089\": container with ID starting with 0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089 not found: ID does not exist" containerID="0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089" Nov 01 15:05:00.355535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.355488 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089"} err="failed to get container status \"0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089\": rpc error: code = NotFound desc = could not find container \"0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089\": container with ID starting with 0c911a89254e7a2ac9a39291f8abf49fa05eeba133e3c7af47d92c4700292089 not found: ID does not exist" Nov 01 15:05:00.449646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449616 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449645 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb918a81-0960-4265-baf2-285161272836-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449656 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449667 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449677 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449687 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449696 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7c6f\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-kube-api-access-h7c6f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449705 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449714 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449723 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ab68bc-e2ea-4021-974b-86ec3260e5a7-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449731 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449739 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449748 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449758 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc886\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-kube-api-access-fc886\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449767 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ab68bc-e2ea-4021-974b-86ec3260e5a7-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449775 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449783 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb918a81-0960-4265-baf2-285161272836-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449792 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb918a81-0960-4265-baf2-285161272836-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449801 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449809 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449818 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgm5\" (UniqueName: \"kubernetes.io/projected/fb918a81-0960-4265-baf2-285161272836-kube-api-access-9fgm5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449826 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/641a2393-367a-44ab-a86f-7b44316c0d9c-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449834 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449844 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef256c70-9de7-4d17-b204-6239b96e14b3-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.449866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.449852 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxlt\" (UniqueName: \"kubernetes.io/projected/641a2393-367a-44ab-a86f-7b44316c0d9c-kube-api-access-rbxlt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:05:00.601396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.601362 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:00.940985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.940891 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22"] Nov 01 15:05:00.944087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:00.944062 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-cvg22"] Nov 01 15:05:01.051945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.051909 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m"] Nov 01 15:05:01.056885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.056852 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-kvn9m"] Nov 01 15:05:01.082412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.082195 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" path="/var/lib/kubelet/pods/90ab68bc-e2ea-4021-974b-86ec3260e5a7/volumes" Nov 01 15:05:01.082913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.082892 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" path="/var/lib/kubelet/pods/ef256c70-9de7-4d17-b204-6239b96e14b3/volumes" Nov 01 15:05:01.108874 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:01.108840 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:05:01.127169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.127132 5473 patch_prober.go:29] interesting pod/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:05:01.127316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.127205 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:05:01.173017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.172987 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7"] Nov 01 15:05:01.192199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.192131 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-wvmt7"] Nov 01 15:05:01.301155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.301112 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl"] Nov 01 15:05:01.306605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:01.306574 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-536a87-openshift-default-7f7cbb4589-5x6gl"] Nov 01 15:05:03.082019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:03.081988 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" path="/var/lib/kubelet/pods/641a2393-367a-44ab-a86f-7b44316c0d9c/volumes" Nov 01 15:05:03.082550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:03.082522 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb918a81-0960-4265-baf2-285161272836" path="/var/lib/kubelet/pods/fb918a81-0960-4265-baf2-285161272836/volumes" Nov 01 15:05:05.079546 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:05.079507 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:09.445900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:09.445865 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:13.913608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:13.913573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:18.234636 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:18.234607 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:18.471883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:18.471853 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:22.971493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:22.971457 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:25.724880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.724843 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7"] Nov 01 15:05:25.725456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725440 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" containerName="discovery" Nov 01 15:05:25.725500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725458 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" containerName="discovery" Nov 01 15:05:25.725500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725488 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" Nov 01 15:05:25.725500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725493 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" Nov 01 15:05:25.725500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725500 5473 cpu_manager.go:414] "containerMap: removing container" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725505 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725512 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725516 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725526 5473 cpu_manager.go:414] "containerMap: removing container" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725530 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725606 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="641a2393-367a-44ab-a86f-7b44316c0d9c" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725619 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98b3fa4-5995-498c-9190-e87eb66aeb4a" containerName="discovery" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725629 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef256c70-9de7-4d17-b204-6239b96e14b3" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725635 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ab68bc-e2ea-4021-974b-86ec3260e5a7" containerName="istio-proxy" Nov 01 15:05:25.725680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.725640 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb918a81-0960-4265-baf2-285161272836" containerName="istio-proxy" Nov 01 15:05:25.729044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.729025 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.782564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.782533 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7"] Nov 01 15:05:25.825132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.825102 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgnk7\" (UniqueName: \"kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.825309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.825191 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.825377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.825344 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.825424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.825384 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.825502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.825487 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.825539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.825522 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.829079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.829058 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-52fffa\"/\"openshift-service-ca.crt\"" Nov 01 15:05:25.829211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.829149 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-52fffa\"/\"default-dockercfg-gmkk7\"" Nov 01 15:05:25.829211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.829153 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-52fffa\"/\"kube-root-ca.crt\"" Nov 01 15:05:25.831608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.831589 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-52fffa\"/\"auth-test-model-52fffa-kserve-self-signed-certs\"" Nov 01 15:05:25.845784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.845762 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:25.926300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926271 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926343 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926371 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926432 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926473 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926589 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgnk7\" (UniqueName: \"kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926718 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926762 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.926896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.926802 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.928876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.928851 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.928979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.928895 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:25.984049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:25.983975 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgnk7\" (UniqueName: \"kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7\") pod \"auth-test-model-52fffa-kserve-844fb56b85-zxnq7\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:26.041438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:26.041405 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:26.411193 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:05:26.411162 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cd91561_4810_47d6_a24c_a6ee5a97d9f8.slice/crio-017d47332fd05b5f0adcdacdb0e4e3a82416c95033f886c0055613d82d188399 WatchSource:0}: Error finding container 017d47332fd05b5f0adcdacdb0e4e3a82416c95033f886c0055613d82d188399: Status 404 returned error can't find the container with id 017d47332fd05b5f0adcdacdb0e4e3a82416c95033f886c0055613d82d188399 Nov 01 15:05:26.414546 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:26.414523 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7"] Nov 01 15:05:26.507814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:26.507778 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" event={"ID":"0cd91561-4810-47d6-a24c-a6ee5a97d9f8","Type":"ContainerStarted","Data":"9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8"} Nov 01 15:05:26.508019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:26.507828 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" event={"ID":"0cd91561-4810-47d6-a24c-a6ee5a97d9f8","Type":"ContainerStarted","Data":"017d47332fd05b5f0adcdacdb0e4e3a82416c95033f886c0055613d82d188399"} Nov 01 15:05:26.625673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:26.625618 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" podStartSLOduration=1.625604155 podStartE2EDuration="1.625604155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:05:26.623516804 +0000 UTC m=+2865.681668655" watchObservedRunningTime="2026-11-01 15:05:26.625604155 +0000 UTC m=+2865.683756005" Nov 01 15:05:27.042398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:27.042369 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:27.352331 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:27.352243 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:27.513945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:27.513892 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:27.808095 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:27.808056 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:31.108116 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:31.108074 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:05:31.995429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:31.995373 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:36.042381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:36.042349 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:36.054959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:36.054938 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:36.509488 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:36.509405 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:36.588612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:36.588580 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:05:40.903030 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:40.902985 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:42.641006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:42.640968 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:42.794408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:42.794359 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:43.439424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:43.439387 5473 scope.go:122] "RemoveContainer" containerID="5c3134a03f7561fa60895d7e2846efbea21dac08ea4407968e8231e7631b7579" Nov 01 15:05:45.256156 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:05:45.256126 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@25-51021-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a04fbb0_db09_4a7e_a4e4_19919e76b584.slice/crio-b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971.scope\": RecentStats: unable to find data in memory cache]" Nov 01 15:05:45.275823 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.275736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:45.645379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.645346 5473 generic.go:356] "Generic (PLEG): container finished" podID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerID="b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971" exitCode=0 Nov 01 15:05:45.645555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.645422 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" event={"ID":"3a04fbb0-db09-4a7e-a4e4-19919e76b584","Type":"ContainerDied","Data":"b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971"} Nov 01 15:05:45.645825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.645804 5473 scope.go:122] "RemoveContainer" containerID="b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971" Nov 01 15:05:45.647360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.647329 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:05:45.647431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.647400 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:05:45.647470 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:45.647433 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:05:46.655586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:46.655553 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" event={"ID":"3a04fbb0-db09-4a7e-a4e4-19919e76b584","Type":"ContainerStarted","Data":"0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b"} Nov 01 15:05:47.284472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:47.284434 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:05:47.284671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:47.284581 5473 patch_prober.go:29] interesting pod/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 15:05:47.284671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:47.284626 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 15:05:47.605402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:47.605321 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:47.767381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:47.767350 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:48.284556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:48.284523 5473 patch_prober.go:29] interesting pod/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 15:05:48.284748 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:48.284588 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 15:05:49.284197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:49.284159 5473 patch_prober.go:29] interesting pod/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 15:05:49.284606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:49.284242 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 15:05:49.656303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:49.647808 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:50.283983 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:50.283949 5473 patch_prober.go:29] interesting pod/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 15:05:50.284169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:50.284005 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 15:05:51.283713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:51.283672 5473 patch_prober.go:29] interesting pod/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Nov 01 15:05:51.284148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:51.283736 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Nov 01 15:05:51.986998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:51.986967 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:05:52.288898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:52.288866 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:05:52.695761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:52.695683 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:05:53.647778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.647732 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7"] Nov 01 15:05:53.656120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.656093 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.705274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.705241 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7"] Nov 01 15:05:53.709646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.709620 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b3f06c\"/\"openshift-service-ca.crt\"" Nov 01 15:05:53.709646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.709632 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b3f06c\"/\"kube-root-ca.crt\"" Nov 01 15:05:53.752023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.751983 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b3f06c\"/\"default-dockercfg-d59pk\"" Nov 01 15:05:53.752023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.751988 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b3f06c\"/\"auth-test-model-b3f06c-kserve-self-signed-certs\"" Nov 01 15:05:53.768709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.768681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:53.777817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.777793 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.778025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.778005 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzsjd\" (UniqueName: \"kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.778185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.778160 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.778278 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.778235 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.778352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.778337 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.778412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.778362 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.878998 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879028 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879067 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879083 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879125 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879241 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzsjd\" (UniqueName: \"kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879552 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879475 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879552 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879511 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.879662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.879643 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.881422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.881399 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.881715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.881694 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.947370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.947306 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzsjd\" (UniqueName: \"kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd\") pod \"auth-test-model-b3f06c-kserve-85768d54d5-472w7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:53.970172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:53.970146 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:54.113332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:54.113290 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:54.339208 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:05:54.339177 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4981ac6e_16d1_4e53_b63b_8066959cecf7.slice/crio-c1494b30e4a134139dd50a20bcb6284d33fbde278883d38246b8bca1276c3bf8 WatchSource:0}: Error finding container c1494b30e4a134139dd50a20bcb6284d33fbde278883d38246b8bca1276c3bf8: Status 404 returned error can't find the container with id c1494b30e4a134139dd50a20bcb6284d33fbde278883d38246b8bca1276c3bf8 Nov 01 15:05:54.342450 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:54.342427 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7"] Nov 01 15:05:54.709108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:54.709020 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" event={"ID":"4981ac6e-16d1-4e53-b63b-8066959cecf7","Type":"ContainerStarted","Data":"ee4c9f22c14c556d58eb2888abd9e955c84e034473f697d97e9b4b252103c082"} Nov 01 15:05:54.709108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:54.709065 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" event={"ID":"4981ac6e-16d1-4e53-b63b-8066959cecf7","Type":"ContainerStarted","Data":"c1494b30e4a134139dd50a20bcb6284d33fbde278883d38246b8bca1276c3bf8"} Nov 01 15:05:54.828704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:54.828658 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" podStartSLOduration=1.8286450840000001 podStartE2EDuration="1.828645084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:05:54.826082941 +0000 UTC m=+2893.884234791" watchObservedRunningTime="2026-11-01 15:05:54.828645084 +0000 UTC m=+2893.886796934" Nov 01 15:05:54.971117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:54.971040 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:05:58.307370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:58.307336 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:05:58.485999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:05:58.485963 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:01.107961 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:06:01.107928 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:06:02.880440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:02.880361 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:03.971021 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:03.970991 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:06:03.983880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:03.983856 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:06:04.788091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:04.788061 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:06:07.323338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:07.323306 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:11.866617 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:11.866568 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:14.151622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:14.151579 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft"] Nov 01 15:06:14.152177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:14.151938 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" containerID="cri-o://da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674" gracePeriod=30 Nov 01 15:06:16.308898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:16.308861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:19.446139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.446110 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 15:06:19.616210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616172 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616246 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616326 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616380 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616440 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616480 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616512 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:06:19.616624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616540 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616572 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616654 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token\") pod \"f26b9161-8159-469b-ac44-718a9632746d\" (UID: \"f26b9161-8159-469b-ac44-718a9632746d\") " Nov 01 15:06:19.616919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616787 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:06:19.616919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616855 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:06:19.617082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.616980 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:06:19.617312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.617288 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.617530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.617513 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.617660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.617645 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.617766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.617754 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.617867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.617680 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:06:19.619041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.619017 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:06:19.619149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.619056 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:06:19.619348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.619264 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:06:19.619348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.619304 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz" pod "f26b9161-8159-469b-ac44-718a9632746d" (UID: "f26b9161-8159-469b-ac44-718a9632746d"). InnerVolumeSpecName "kube-api-access-42lsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:06:19.719348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.719037 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26b9161-8159-469b-ac44-718a9632746d-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.719348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.719137 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26b9161-8159-469b-ac44-718a9632746d-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.719348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.719156 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26b9161-8159-469b-ac44-718a9632746d-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.719348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.719237 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lsz\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-kube-api-access-42lsz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.719348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.719261 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26b9161-8159-469b-ac44-718a9632746d-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:06:19.915326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.915295 5473 generic.go:356] "Generic (PLEG): container finished" podID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerID="699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df" exitCode=0 Nov 01 15:06:19.915511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.915375 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" event={"ID":"43b62e39-2b10-41d3-80e5-a449d34d0f23","Type":"ContainerDied","Data":"699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df"} Nov 01 15:06:19.915864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.915849 5473 scope.go:122] "RemoveContainer" containerID="699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df" Nov 01 15:06:19.916848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.916821 5473 generic.go:356] "Generic (PLEG): container finished" podID="f26b9161-8159-469b-ac44-718a9632746d" containerID="da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674" exitCode=0 Nov 01 15:06:19.916960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.916872 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" event={"ID":"f26b9161-8159-469b-ac44-718a9632746d","Type":"ContainerDied","Data":"da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674"} Nov 01 15:06:19.916960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.916875 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" Nov 01 15:06:19.916960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.916904 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft" event={"ID":"f26b9161-8159-469b-ac44-718a9632746d","Type":"ContainerDied","Data":"6949ec8ca32fb79da2371c3d036735d914b8abe3b2f810f7086439055e577bcf"} Nov 01 15:06:19.916960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.916923 5473 scope.go:122] "RemoveContainer" containerID="da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674" Nov 01 15:06:19.917608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.917582 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:06:19.917709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.917665 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:06:19.917785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.917717 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:06:19.931110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.931092 5473 scope.go:122] "RemoveContainer" containerID="da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674" Nov 01 15:06:19.931409 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:06:19.931386 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674\": container with ID starting with da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674 not found: ID does not exist" containerID="da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674" Nov 01 15:06:19.931504 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:19.931412 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674"} err="failed to get container status \"da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674\": rpc error: code = NotFound desc = could not find container \"da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674\": container with ID starting with da587d79dded241df3f797c0c7dc32ff7a533da452f33d284d6329d81a376674 not found: ID does not exist" Nov 01 15:06:20.150948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:20.150915 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft"] Nov 01 15:06:20.153643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:20.153620 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-007270-openshift-default-65777bf9df-w58ft"] Nov 01 15:06:20.702686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:20.702655 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:20.951153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:20.951116 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" event={"ID":"43b62e39-2b10-41d3-80e5-a449d34d0f23","Type":"ContainerStarted","Data":"7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3"} Nov 01 15:06:21.082588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:21.082554 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26b9161-8159-469b-ac44-718a9632746d" path="/var/lib/kubelet/pods/f26b9161-8159-469b-ac44-718a9632746d/volumes" Nov 01 15:06:21.708160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:21.708124 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:06:21.709697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:21.709671 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 15:06:21.709816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:21.709721 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 15:06:22.707748 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:22.707713 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 15:06:22.707928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:22.707777 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 15:06:23.707889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:23.707847 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 15:06:23.708333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:23.707910 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 15:06:24.708273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:24.708237 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 15:06:24.708811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:24.708295 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 15:06:25.145919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:25.145769 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:25.707869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:25.707833 5473 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" start-of-body= Nov 01 15:06:25.708057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:25.707887 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.233:15021/healthz/ready\": dial tcp 10.134.0.233:15021: connect: connection refused" Nov 01 15:06:26.020497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:26.020464 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:06:26.245545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:26.245511 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:26.712569 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:26.712541 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:06:26.997844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:26.997773 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:06:29.837582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:29.837539 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:31.108282 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:06:31.108247 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:06:34.313314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:34.313275 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:35.477657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:35.477624 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:35.921905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:35.921871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:36.535902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.535866 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5"] Nov 01 15:06:36.536447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.536433 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" Nov 01 15:06:36.536515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.536448 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" Nov 01 15:06:36.536562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.536552 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26b9161-8159-469b-ac44-718a9632746d" containerName="istio-proxy" Nov 01 15:06:36.541085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.541068 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.593885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.593839 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-8fd363-openshift-default-dockercfg-p6rfq\"" Nov 01 15:06:36.635342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635304 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635349 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635561 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635610 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rwvf\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635688 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635716 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635821 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.635940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635926 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.636004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.635954 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.649151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.649124 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5"] Nov 01 15:06:36.667857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.667812 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:36.737400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737363 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737399 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737440 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737462 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737539 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737556 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rwvf\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737582 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737599 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737626 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737925 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.737968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.737950 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.738130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.738103 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.738244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.738200 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.738482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.738463 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.740043 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.740017 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.740043 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.740030 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.795368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.795309 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.795683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.795662 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rwvf\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf\") pod \"e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.855253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.855205 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:36.866230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:36.866180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:37.281338 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:06:37.281303 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c04e05_8282_4efc_bf59_f6055d051095.slice/crio-5846ec2f16cc89cb90b3e3054192aa0e2f73b1d20bf41f730d9c7c7f0bf02bc2 WatchSource:0}: Error finding container 5846ec2f16cc89cb90b3e3054192aa0e2f73b1d20bf41f730d9c7c7f0bf02bc2: Status 404 returned error can't find the container with id 5846ec2f16cc89cb90b3e3054192aa0e2f73b1d20bf41f730d9c7c7f0bf02bc2 Nov 01 15:06:37.283835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:37.283801 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:06:37.283929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:37.283871 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:06:37.283929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:37.283900 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:06:37.284077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:37.284057 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5"] Nov 01 15:06:38.073994 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:38.073956 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" event={"ID":"a1c04e05-8282-4efc-bf59-f6055d051095","Type":"ContainerStarted","Data":"cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16"} Nov 01 15:06:38.074417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:38.074012 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" event={"ID":"a1c04e05-8282-4efc-bf59-f6055d051095","Type":"ContainerStarted","Data":"5846ec2f16cc89cb90b3e3054192aa0e2f73b1d20bf41f730d9c7c7f0bf02bc2"} Nov 01 15:06:38.242615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:38.242566 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podStartSLOduration=2.242553052 podStartE2EDuration="2.242553052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:06:38.240155973 +0000 UTC m=+2937.298307825" watchObservedRunningTime="2026-11-01 15:06:38.242553052 +0000 UTC m=+2937.300704898" Nov 01 15:06:38.855495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:38.855458 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:38.861424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:38.861401 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:39.081514 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:39.081487 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:39.082154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:39.082132 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:06:39.085758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:39.085737 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:41.084785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:41.084753 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:41.275393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:41.275358 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:43.505623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:43.505593 5473 scope.go:122] "RemoveContainer" containerID="ec88fb90ae8ad3bcd0e7d6a1f686f61dc6f4103fdc521b45839409fdb4e667d7" Nov 01 15:06:43.991374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:43.991344 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:44.166128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.166091 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:44.335320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.335286 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:44.557378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.557341 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk"] Nov 01 15:06:44.562418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.562399 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.614880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.614798 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk"] Nov 01 15:06:44.660815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.660782 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8fd363-dockercfg-jw9pt\"" Nov 01 15:06:44.710855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.710821 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8fd363\"" Nov 01 15:06:44.714728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.714694 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t"] Nov 01 15:06:44.720044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.720023 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:44.731632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.731610 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxw7t\" (UniqueName: \"kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t\") pod \"payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.731753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.731659 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.768621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.768600 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t"] Nov 01 15:06:44.833196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.833167 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn\") pod \"payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:44.833374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.833257 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:44.833427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.833412 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxw7t\" (UniqueName: \"kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t\") pod \"payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.833482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.833466 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.834269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.834241 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.891437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.891374 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxw7t\" (UniqueName: \"kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t\") pod \"payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:44.934905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.934876 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn\") pod \"payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:44.935061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.934924 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:44.935619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.935600 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:44.991808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:44.991782 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn\") pod \"payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:45.034125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:45.034100 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:45.176661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:45.176590 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:45.409074 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:45.408698 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t"] Nov 01 15:06:45.542303 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:06:45.542259 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73aa7734_f8c0_41ac_bfa9_a58918e1c6e6.slice/crio-597175bfb1e5de9a6f4a5c2b09fdc7c38f3df7eafeec3f532d60a4724f1bd8b3 WatchSource:0}: Error finding container 597175bfb1e5de9a6f4a5c2b09fdc7c38f3df7eafeec3f532d60a4724f1bd8b3: Status 404 returned error can't find the container with id 597175bfb1e5de9a6f4a5c2b09fdc7c38f3df7eafeec3f532d60a4724f1bd8b3 Nov 01 15:06:45.544728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:45.544702 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk"] Nov 01 15:06:46.129783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.129741 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" event={"ID":"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6","Type":"ContainerStarted","Data":"e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda"} Nov 01 15:06:46.130308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.129797 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" event={"ID":"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6","Type":"ContainerStarted","Data":"597175bfb1e5de9a6f4a5c2b09fdc7c38f3df7eafeec3f532d60a4724f1bd8b3"} Nov 01 15:06:46.130308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.129824 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:46.131382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.131350 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" event={"ID":"95b41ae1-1984-4103-8f6d-9308335bbc95","Type":"ContainerStarted","Data":"01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1"} Nov 01 15:06:46.131561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.131403 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" event={"ID":"95b41ae1-1984-4103-8f6d-9308335bbc95","Type":"ContainerStarted","Data":"eee1c60e886f03f51dd1daf48cc5759237e67a0dd24c5394359503a0d76b3142"} Nov 01 15:06:46.131561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.131444 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:46.243635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.243580 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" podStartSLOduration=2.243568018 podStartE2EDuration="2.243568018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:06:46.243266743 +0000 UTC m=+2945.301418592" watchObservedRunningTime="2026-11-01 15:06:46.243568018 +0000 UTC m=+2945.301719869" Nov 01 15:06:46.361937 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:46.361857 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" podStartSLOduration=2.36184206 podStartE2EDuration="2.36184206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:06:46.357262052 +0000 UTC m=+2945.415413926" watchObservedRunningTime="2026-11-01 15:06:46.36184206 +0000 UTC m=+2945.419993909" Nov 01 15:06:48.548940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:48.548911 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:49.391303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.391266 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc"] Nov 01 15:06:49.398138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.398113 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.444115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.444086 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc"] Nov 01 15:06:49.491201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.491176 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7794bf\"/\"kube-root-ca.crt\"" Nov 01 15:06:49.491387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.491277 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7794bf\"/\"default-dockercfg-8kzg8\"" Nov 01 15:06:49.496015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.495986 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.496166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.496143 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.496283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.496268 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrmq\" (UniqueName: \"kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.496332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.496295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.496332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.496313 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.496485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.496464 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.500647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.500627 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-7794bf\"/\"openshift-service-ca.crt\"" Nov 01 15:06:49.500769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.500690 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-7794bf\"/\"explicit-model-7794bf-kserve-self-signed-certs\"" Nov 01 15:06:49.503420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.503396 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:49.597989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.597961 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598049 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrmq\" (UniqueName: \"kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598079 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598099 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598164 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598206 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598608 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598675 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.598768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.598728 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.600410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.600390 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.600701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.600683 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.656423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.656357 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrmq\" (UniqueName: \"kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq\") pod \"explicit-model-7794bf-kserve-ccc8f6985-xs7lc\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:49.712980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:49.712951 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:50.080656 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:06:50.080629 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb009e801_1a1f_43e1_a3e4_0210edf2bcf1.slice/crio-ca868fc58f66b7277a9fb821a87839fe49152dede98b41fc35acd1e6566affbe WatchSource:0}: Error finding container ca868fc58f66b7277a9fb821a87839fe49152dede98b41fc35acd1e6566affbe: Status 404 returned error can't find the container with id ca868fc58f66b7277a9fb821a87839fe49152dede98b41fc35acd1e6566affbe Nov 01 15:06:50.083897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:50.083874 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc"] Nov 01 15:06:50.158668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:50.158636 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" event={"ID":"b009e801-1a1f-43e1-a3e4-0210edf2bcf1","Type":"ContainerStarted","Data":"1f1f44ef8bad6049060974edaf95df7867649413a5d26847e526298a4d281f16"} Nov 01 15:06:50.158783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:50.158684 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" event={"ID":"b009e801-1a1f-43e1-a3e4-0210edf2bcf1","Type":"ContainerStarted","Data":"ca868fc58f66b7277a9fb821a87839fe49152dede98b41fc35acd1e6566affbe"} Nov 01 15:06:50.277127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:50.277073 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" podStartSLOduration=1.27705903 podStartE2EDuration="1.27705903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:06:50.273867463 +0000 UTC m=+2949.332019312" watchObservedRunningTime="2026-11-01 15:06:50.27705903 +0000 UTC m=+2949.335210881" Nov 01 15:06:50.713612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:50.713578 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:52.992689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:52.992655 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:57.140729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:57.140700 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:06:57.141228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:57.140778 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:06:57.507901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:57.507820 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:59.189553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:59.189522 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:06:59.713363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:59.713332 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:06:59.725909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:06:59.725885 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:07:00.232257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:00.232204 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:07:01.109807 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:07:01.109774 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:07:01.919074 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:01.919042 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:06.398483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:06.398449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:07.919651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:07.919608 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:08.135490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:08.135455 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:10.800170 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:10.800138 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:15.225811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:15.225769 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:19.658559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:19.658495 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:24.174338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:24.174258 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:28.512849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:28.512816 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:29.413067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:29.413029 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:31.107414 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:07:31.107382 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:07:33.039036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:33.039002 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:34.387869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:34.387832 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc"] Nov 01 15:07:34.606381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:34.606345 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:37.627149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:37.627109 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:42.107888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:42.107858 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:44.047897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:44.047866 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:07:44.050632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:44.050609 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:07:45.482446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:45.482411 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:46.469004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:46.468961 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:46.689517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:46.689484 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:49.409729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.409682 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" podUID="b009e801-1a1f-43e1-a3e4-0210edf2bcf1" containerName="main" containerID="cri-o://1f1f44ef8bad6049060974edaf95df7867649413a5d26847e526298a4d281f16" gracePeriod=45 Nov 01 15:07:49.603369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.603336 5473 generic.go:356] "Generic (PLEG): container finished" podID="b009e801-1a1f-43e1-a3e4-0210edf2bcf1" containerID="1f1f44ef8bad6049060974edaf95df7867649413a5d26847e526298a4d281f16" exitCode=0 Nov 01 15:07:49.603533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.603376 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" event={"ID":"b009e801-1a1f-43e1-a3e4-0210edf2bcf1","Type":"ContainerDied","Data":"1f1f44ef8bad6049060974edaf95df7867649413a5d26847e526298a4d281f16"} Nov 01 15:07:49.671048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.670991 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:07:49.724393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724363 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq\" (UniqueName: \"kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq\") pod \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " Nov 01 15:07:49.724579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724407 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir\") pod \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " Nov 01 15:07:49.724579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724447 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home\") pod \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " Nov 01 15:07:49.724712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724581 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache\") pod \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " Nov 01 15:07:49.724712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724608 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm\") pod \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " Nov 01 15:07:49.724712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724645 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs\") pod \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\" (UID: \"b009e801-1a1f-43e1-a3e4-0210edf2bcf1\") " Nov 01 15:07:49.724712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724685 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir" pod "b009e801-1a1f-43e1-a3e4-0210edf2bcf1" (UID: "b009e801-1a1f-43e1-a3e4-0210edf2bcf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:07:49.724886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724795 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home" pod "b009e801-1a1f-43e1-a3e4-0210edf2bcf1" (UID: "b009e801-1a1f-43e1-a3e4-0210edf2bcf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:07:49.724886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.724811 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache" pod "b009e801-1a1f-43e1-a3e4-0210edf2bcf1" (UID: "b009e801-1a1f-43e1-a3e4-0210edf2bcf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:07:49.725271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.725208 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:07:49.725271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.725272 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:07:49.725456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.725281 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:07:49.726713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.726691 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm" pod "b009e801-1a1f-43e1-a3e4-0210edf2bcf1" (UID: "b009e801-1a1f-43e1-a3e4-0210edf2bcf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:07:49.726847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.726828 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs" pod "b009e801-1a1f-43e1-a3e4-0210edf2bcf1" (UID: "b009e801-1a1f-43e1-a3e4-0210edf2bcf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:07:49.726892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.726850 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq" pod "b009e801-1a1f-43e1-a3e4-0210edf2bcf1" (UID: "b009e801-1a1f-43e1-a3e4-0210edf2bcf1"). InnerVolumeSpecName "kube-api-access-csrmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:07:49.826813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.826782 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:07:49.826813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.826815 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:07:49.827019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:49.826830 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrmq\" (UniqueName: \"kubernetes.io/projected/b009e801-1a1f-43e1-a3e4-0210edf2bcf1-kube-api-access-csrmq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:07:50.606342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.606311 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:50.611328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.611307 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" Nov 01 15:07:50.611478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.611344 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc" event={"ID":"b009e801-1a1f-43e1-a3e4-0210edf2bcf1","Type":"ContainerDied","Data":"ca868fc58f66b7277a9fb821a87839fe49152dede98b41fc35acd1e6566affbe"} Nov 01 15:07:50.611478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.611400 5473 scope.go:122] "RemoveContainer" containerID="1f1f44ef8bad6049060974edaf95df7867649413a5d26847e526298a4d281f16" Nov 01 15:07:50.776798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.776762 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc"] Nov 01 15:07:50.778126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.778098 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:50.826716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:50.826683 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-7794bf/explicit-model-7794bf-kserve-ccc8f6985-xs7lc"] Nov 01 15:07:51.083008 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:51.082977 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b009e801-1a1f-43e1-a3e4-0210edf2bcf1" path="/var/lib/kubelet/pods/b009e801-1a1f-43e1-a3e4-0210edf2bcf1/volumes" Nov 01 15:07:51.102183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:51.102147 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:55.543831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:55.543799 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:58.990395 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:58.990358 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:07:59.895962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:07:59.895916 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:00.305149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:00.305115 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:01.116002 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:08:01.115950 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:08:04.293768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:04.293733 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:08.680781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:08.680740 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:10.658568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:10.658533 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q"] Nov 01 15:08:11.023809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:11.023776 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:11.191019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:11.190991 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:11.747188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:11.747153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:13.310352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:13.310320 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:15.008082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:15.008045 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:15.659688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:15.659640 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" podUID="04bacb08-bc51-462e-8b71-8f35b201a6a5" containerName="payload-processing" containerID="cri-o://ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3" gracePeriod=25 Nov 01 15:08:15.930192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:15.930169 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:08:16.045482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.045443 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume\") pod \"04bacb08-bc51-462e-8b71-8f35b201a6a5\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " Nov 01 15:08:16.045957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.045514 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq\" (UniqueName: \"kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq\") pod \"04bacb08-bc51-462e-8b71-8f35b201a6a5\" (UID: \"04bacb08-bc51-462e-8b71-8f35b201a6a5\") " Nov 01 15:08:16.045957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.045866 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume" pod "04bacb08-bc51-462e-8b71-8f35b201a6a5" (UID: "04bacb08-bc51-462e-8b71-8f35b201a6a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:08:16.046504 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.046477 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04bacb08-bc51-462e-8b71-8f35b201a6a5-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:16.047887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.047859 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq" pod "04bacb08-bc51-462e-8b71-8f35b201a6a5" (UID: "04bacb08-bc51-462e-8b71-8f35b201a6a5"). InnerVolumeSpecName "kube-api-access-qx4kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:08:16.147618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.147584 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4kq\" (UniqueName: \"kubernetes.io/projected/04bacb08-bc51-462e-8b71-8f35b201a6a5-kube-api-access-qx4kq\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:16.780813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.780777 5473 generic.go:356] "Generic (PLEG): container finished" podID="04bacb08-bc51-462e-8b71-8f35b201a6a5" containerID="ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3" exitCode=0 Nov 01 15:08:16.780813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.780811 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" Nov 01 15:08:16.781051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.780887 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" event={"ID":"04bacb08-bc51-462e-8b71-8f35b201a6a5","Type":"ContainerDied","Data":"ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3"} Nov 01 15:08:16.781051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.780937 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q" event={"ID":"04bacb08-bc51-462e-8b71-8f35b201a6a5","Type":"ContainerDied","Data":"e35a56a0b79ad742f6691fbfb33a89737e2292d489386c403e41ebb17f8bb84c"} Nov 01 15:08:16.781051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.780969 5473 scope.go:122] "RemoveContainer" containerID="ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3" Nov 01 15:08:16.791813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.791797 5473 scope.go:122] "RemoveContainer" containerID="ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3" Nov 01 15:08:16.792117 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:08:16.792098 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3\": container with ID starting with ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3 not found: ID does not exist" containerID="ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3" Nov 01 15:08:16.792174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.792135 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3"} err="failed to get container status \"ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3\": rpc error: code = NotFound desc = could not find container \"ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3\": container with ID starting with ac48c4538b327ac4e135f490733a27c6210f2e79d6a87327ab889450f0eccbf3 not found: ID does not exist" Nov 01 15:08:16.946739 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.946707 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q"] Nov 01 15:08:16.996091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:16.996060 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-7794bf-75bcd686d7-4tz8q"] Nov 01 15:08:17.091387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:17.091308 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bacb08-bc51-462e-8b71-8f35b201a6a5" path="/var/lib/kubelet/pods/04bacb08-bc51-462e-8b71-8f35b201a6a5/volumes" Nov 01 15:08:17.403758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:17.403677 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:17.813304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:17.813277 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:22.213489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:22.213455 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:26.799650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:26.799614 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:30.650763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:30.650723 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:31.110885 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:08:31.110850 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:08:32.025567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.025537 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w"] Nov 01 15:08:32.026113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.026100 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b009e801-1a1f-43e1-a3e4-0210edf2bcf1" containerName="main" Nov 01 15:08:32.026159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.026115 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b009e801-1a1f-43e1-a3e4-0210edf2bcf1" containerName="main" Nov 01 15:08:32.026159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.026140 5473 cpu_manager.go:414] "containerMap: removing container" podUID="04bacb08-bc51-462e-8b71-8f35b201a6a5" containerName="payload-processing" Nov 01 15:08:32.026159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.026144 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bacb08-bc51-462e-8b71-8f35b201a6a5" containerName="payload-processing" Nov 01 15:08:32.026276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.026231 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bacb08-bc51-462e-8b71-8f35b201a6a5" containerName="payload-processing" Nov 01 15:08:32.026276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.026247 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b009e801-1a1f-43e1-a3e4-0210edf2bcf1" containerName="main" Nov 01 15:08:32.031230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.031204 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.083066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.083028 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w"] Nov 01 15:08:32.128727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.128692 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:08:32.128727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.128701 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:08:32.164143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.164111 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrmf\" (UniqueName: \"kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.164336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.164259 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.164336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.164320 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.255291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.255249 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:32.265799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.265760 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.265955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.265876 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.265955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.265943 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrmf\" (UniqueName: \"kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.266669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.266636 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.268207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.268183 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.323531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.323466 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrmf\" (UniqueName: \"kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf\") pod \"maas-api-revoke-sub-68d7e5f4-ts69w\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.346123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.346094 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:32.433104 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.432342 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:32.724699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.724664 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w"] Nov 01 15:08:32.907738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:32.907698 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" event={"ID":"c567c7a4-5dbb-426f-9721-3f084d24b005","Type":"ContainerStarted","Data":"223e59f8e04312232e518c5e64ff346740c60710912d9b87f1e3f0e12d793d77"} Nov 01 15:08:33.786495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:33.786465 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:33.916859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:33.916777 5473 generic.go:356] "Generic (PLEG): container finished" podID="c567c7a4-5dbb-426f-9721-3f084d24b005" containerID="ccd09a466e8f08164581cad3690f2d528e0bee52dc54cc3de8c4f43832e1de14" exitCode=0 Nov 01 15:08:33.917013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:33.916865 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" event={"ID":"c567c7a4-5dbb-426f-9721-3f084d24b005","Type":"ContainerDied","Data":"ccd09a466e8f08164581cad3690f2d528e0bee52dc54cc3de8c4f43832e1de14"} Nov 01 15:08:34.299969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.299936 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6"] Nov 01 15:08:34.300230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.300191 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" containerID="cri-o://0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b" gracePeriod=30 Nov 01 15:08:34.358435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.358402 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:34.925441 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.925416 5473 generic.go:356] "Generic (PLEG): container finished" podID="a1c04e05-8282-4efc-bf59-f6055d051095" containerID="cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16" exitCode=0 Nov 01 15:08:34.925801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.925489 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" event={"ID":"a1c04e05-8282-4efc-bf59-f6055d051095","Type":"ContainerDied","Data":"cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16"} Nov 01 15:08:34.925965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.925949 5473 scope.go:122] "RemoveContainer" containerID="cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16" Nov 01 15:08:34.927429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.927400 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:08:34.927521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.927473 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:08:34.927521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:34.927505 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:08:35.014481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.014442 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:35.073003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.072967 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:35.204343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.203939 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf\" (UniqueName: \"kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf\") pod \"c567c7a4-5dbb-426f-9721-3f084d24b005\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " Nov 01 15:08:35.204343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.204125 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca\") pod \"c567c7a4-5dbb-426f-9721-3f084d24b005\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " Nov 01 15:08:35.204343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.204191 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca\") pod \"c567c7a4-5dbb-426f-9721-3f084d24b005\" (UID: \"c567c7a4-5dbb-426f-9721-3f084d24b005\") " Nov 01 15:08:35.205262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.205188 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca" pod "c567c7a4-5dbb-426f-9721-3f084d24b005" (UID: "c567c7a4-5dbb-426f-9721-3f084d24b005"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:08:35.207126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.207103 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca" pod "c567c7a4-5dbb-426f-9721-3f084d24b005" (UID: "c567c7a4-5dbb-426f-9721-3f084d24b005"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:08:35.212731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.212559 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf" pod "c567c7a4-5dbb-426f-9721-3f084d24b005" (UID: "c567c7a4-5dbb-426f-9721-3f084d24b005"). InnerVolumeSpecName "kube-api-access-grrmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:08:35.305472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.305436 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grrmf\" (UniqueName: \"kubernetes.io/projected/c567c7a4-5dbb-426f-9721-3f084d24b005-kube-api-access-grrmf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:35.305472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.305465 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:35.305472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.305476 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c567c7a4-5dbb-426f-9721-3f084d24b005-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:35.823645 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.823609 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d"] Nov 01 15:08:35.844241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.824294 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c567c7a4-5dbb-426f-9721-3f084d24b005" containerName="revoke-keys" Nov 01 15:08:35.844520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.844493 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c567c7a4-5dbb-426f-9721-3f084d24b005" containerName="revoke-keys" Nov 01 15:08:35.844934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.844915 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c567c7a4-5dbb-426f-9721-3f084d24b005" containerName="revoke-keys" Nov 01 15:08:35.851238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.850638 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.890342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.890267 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d"] Nov 01 15:08:35.912413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912382 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912424 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912444 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912526 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912605 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912643 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912771 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5wz\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912821 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.912962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.912946 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:35.942527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.942490 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-db6f2e-openshift-default-dockercfg-npqf9\"" Nov 01 15:08:35.952281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.948148 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:35.957938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.957905 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" event={"ID":"c567c7a4-5dbb-426f-9721-3f084d24b005","Type":"ContainerDied","Data":"223e59f8e04312232e518c5e64ff346740c60710912d9b87f1e3f0e12d793d77"} Nov 01 15:08:35.958098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.957964 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="223e59f8e04312232e518c5e64ff346740c60710912d9b87f1e3f0e12d793d77" Nov 01 15:08:35.958098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.957918 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w" Nov 01 15:08:35.960226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:35.960191 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" event={"ID":"a1c04e05-8282-4efc-bf59-f6055d051095","Type":"ContainerStarted","Data":"dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653"} Nov 01 15:08:36.014639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014606 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.014811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014654 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.014811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014682 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.014811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014712 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.014811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014777 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.014960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014816 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.014998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.014964 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5wz\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015017 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015130 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015266 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015392 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015480 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015541 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.015811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.015661 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.017302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.017282 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.017647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.017632 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.071961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.071924 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.072284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.072260 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5wz\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz\") pod \"e2e-noten-db6f2e-openshift-default-f685796f9-s776d\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.176170 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.176070 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:36.204359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.204331 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:36.616201 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:08:36.616167 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ee1935_935d_414a_a848_0c0ce0f607d4.slice/crio-ed9f52813b933105e484afa83f20884df9e5a4e2da6fd69d758de590dbf9b96f WatchSource:0}: Error finding container ed9f52813b933105e484afa83f20884df9e5a4e2da6fd69d758de590dbf9b96f: Status 404 returned error can't find the container with id ed9f52813b933105e484afa83f20884df9e5a4e2da6fd69d758de590dbf9b96f Nov 01 15:08:36.619080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.619045 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:08:36.619198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.619128 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:08:36.619198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.619172 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:08:36.619836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.619818 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d"] Nov 01 15:08:36.855543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.855508 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:08:36.856776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.856750 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 15:08:36.856914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.856804 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 15:08:36.974775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.974682 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" event={"ID":"a1ee1935-935d-414a-a848-0c0ce0f607d4","Type":"ContainerStarted","Data":"8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c"} Nov 01 15:08:36.974775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:36.974735 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" event={"ID":"a1ee1935-935d-414a-a848-0c0ce0f607d4","Type":"ContainerStarted","Data":"ed9f52813b933105e484afa83f20884df9e5a4e2da6fd69d758de590dbf9b96f"} Nov 01 15:08:37.094836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.094769 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" podStartSLOduration=2.09475339 podStartE2EDuration="2.09475339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:08:37.091041174 +0000 UTC m=+3056.149193022" watchObservedRunningTime="2026-11-01 15:08:37.09475339 +0000 UTC m=+3056.152905240" Nov 01 15:08:37.102868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.102834 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:37.177181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.177149 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:37.183417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.183391 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:37.611706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.611672 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:37.696794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.696762 5473 patch_prober.go:29] interesting pod/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:08:37.696959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.696816 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:08:37.816347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.816309 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:37.856230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.856183 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 15:08:37.856413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.856255 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 15:08:37.979913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.979822 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:37.981832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.981815 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:37.982779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:37.982756 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:08:38.855781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:38.855749 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 15:08:38.855984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:38.855801 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 15:08:39.081811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.081782 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:08:39.607621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.607598 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:08:39.690799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.690761 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.690999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.690826 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.690999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.690940 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691004 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691029 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691081 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691232 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691276 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691315 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket\") pod \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\" (UID: \"3a04fbb0-db09-4a7e-a4e4-19919e76b584\") " Nov 01 15:08:39.691486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691422 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:08:39.691486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691466 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:08:39.691662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691641 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:08:39.691850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691815 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:08:39.691896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.691857 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:08:39.692199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.692180 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.692270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.692205 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.692270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.692236 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.692270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.692251 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.692270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.692266 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.693511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.693471 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:08:39.693620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.693521 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:08:39.693620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.693546 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:08:39.694116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.694096 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh" pod "3a04fbb0-db09-4a7e-a4e4-19919e76b584" (UID: "3a04fbb0-db09-4a7e-a4e4-19919e76b584"). InnerVolumeSpecName "kube-api-access-m2mrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:08:39.796264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.796198 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.796264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.796252 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.796264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.796269 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mrh\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-kube-api-access-m2mrh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.796587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.796285 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a04fbb0-db09-4a7e-a4e4-19919e76b584-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:39.856279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.856180 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 15:08:39.856279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.856252 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 15:08:39.997970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.997936 5473 generic.go:356] "Generic (PLEG): container finished" podID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerID="0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b" exitCode=0 Nov 01 15:08:39.998131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.997967 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" event={"ID":"3a04fbb0-db09-4a7e-a4e4-19919e76b584","Type":"ContainerDied","Data":"0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b"} Nov 01 15:08:39.998131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.997997 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" Nov 01 15:08:39.998131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.998037 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6" event={"ID":"3a04fbb0-db09-4a7e-a4e4-19919e76b584","Type":"ContainerDied","Data":"f2b0deb50af89ad2b9943a45047c7b99de2d336eca6d4e63a93958d1dd9aa80f"} Nov 01 15:08:39.998131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:39.998060 5473 scope.go:122] "RemoveContainer" containerID="0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b" Nov 01 15:08:40.010494 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.010480 5473 scope.go:122] "RemoveContainer" containerID="b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971" Nov 01 15:08:40.020372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.020345 5473 scope.go:122] "RemoveContainer" containerID="0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b" Nov 01 15:08:40.020668 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:08:40.020644 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b\": container with ID starting with 0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b not found: ID does not exist" containerID="0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b" Nov 01 15:08:40.020753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.020680 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b"} err="failed to get container status \"0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b\": rpc error: code = NotFound desc = could not find container \"0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b\": container with ID starting with 0d4cfdd78e364a6a0d884ec5c108e6f4f39568b3029fe06fba25e6394152ad8b not found: ID does not exist" Nov 01 15:08:40.020753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.020703 5473 scope.go:122] "RemoveContainer" containerID="b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971" Nov 01 15:08:40.020979 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:08:40.020959 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971\": container with ID starting with b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971 not found: ID does not exist" containerID="b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971" Nov 01 15:08:40.021018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.020986 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971"} err="failed to get container status \"b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971\": rpc error: code = NotFound desc = could not find container \"b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971\": container with ID starting with b519df19c22d8168a602c1dfb66a8421532c4432fe57723be7ae1e7e42bcd971 not found: ID does not exist" Nov 01 15:08:40.165118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.165034 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6"] Nov 01 15:08:40.169921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.169898 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7794bf-openshift-default-5b985dcc97-djth6"] Nov 01 15:08:40.856175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.856138 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 01 15:08:40.856370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:40.856199 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 01 15:08:41.081521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:41.081481 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" path="/var/lib/kubelet/pods/3a04fbb0-db09-4a7e-a4e4-19919e76b584/volumes" Nov 01 15:08:41.765105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:41.765075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:41.859864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:41.859837 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:08:41.860893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:41.860874 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:08:42.713666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.713636 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp"] Nov 01 15:08:42.714167 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.714155 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" Nov 01 15:08:42.714235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.714168 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" Nov 01 15:08:42.714235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.714184 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" Nov 01 15:08:42.714235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.714189 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" Nov 01 15:08:42.714372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.714316 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" Nov 01 15:08:42.714372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.714328 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a04fbb0-db09-4a7e-a4e4-19919e76b584" containerName="istio-proxy" Nov 01 15:08:42.717710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.717693 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.770330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.770303 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:08:42.770770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.770445 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:08:42.771950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.771929 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp"] Nov 01 15:08:42.835668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.835642 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.835805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.835690 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.835805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.835784 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxgt\" (UniqueName: \"kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.936891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.936859 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxgt\" (UniqueName: \"kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.937072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.936971 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.937072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.937007 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.937671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.937651 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.939326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.939304 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:42.993777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:42.993720 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxgt\" (UniqueName: \"kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt\") pod \"maas-api-revoke-sub-ef7ec078-kdgrp\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:43.030931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:43.030897 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:43.398705 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:08:43.398665 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c19139_3cec_45bf_9911_586887b31e85.slice/crio-d665252e79fc7a49cf1f2cb4b96e4aba102ff21f16e4f7d5df233a615050c421 WatchSource:0}: Error finding container d665252e79fc7a49cf1f2cb4b96e4aba102ff21f16e4f7d5df233a615050c421: Status 404 returned error can't find the container with id d665252e79fc7a49cf1f2cb4b96e4aba102ff21f16e4f7d5df233a615050c421 Nov 01 15:08:43.400947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:43.400924 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp"] Nov 01 15:08:44.027283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:44.027245 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" event={"ID":"37c19139-3cec-45bf-9911-586887b31e85","Type":"ContainerStarted","Data":"0c807e71df9dd772fa5cccd69c1ca04ce2b7301d152ed027f08991a80b9ab7ad"} Nov 01 15:08:44.027644 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:44.027331 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" event={"ID":"37c19139-3cec-45bf-9911-586887b31e85","Type":"ContainerStarted","Data":"d665252e79fc7a49cf1f2cb4b96e4aba102ff21f16e4f7d5df233a615050c421"} Nov 01 15:08:44.140857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:44.140807 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" podStartSLOduration=2.140794296 podStartE2EDuration="2.140794296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:08:44.140070372 +0000 UTC m=+3063.198222222" watchObservedRunningTime="2026-11-01 15:08:44.140794296 +0000 UTC m=+3063.198946145" Nov 01 15:08:45.035853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:45.035816 5473 generic.go:356] "Generic (PLEG): container finished" podID="37c19139-3cec-45bf-9911-586887b31e85" containerID="0c807e71df9dd772fa5cccd69c1ca04ce2b7301d152ed027f08991a80b9ab7ad" exitCode=0 Nov 01 15:08:45.036303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:45.035897 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" event={"ID":"37c19139-3cec-45bf-9911-586887b31e85","Type":"ContainerDied","Data":"0c807e71df9dd772fa5cccd69c1ca04ce2b7301d152ed027f08991a80b9ab7ad"} Nov 01 15:08:46.197115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.197088 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:46.197787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.197761 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:46.378741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.378659 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca\") pod \"37c19139-3cec-45bf-9911-586887b31e85\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " Nov 01 15:08:46.378914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.378783 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt\" (UniqueName: \"kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt\") pod \"37c19139-3cec-45bf-9911-586887b31e85\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " Nov 01 15:08:46.378986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.378932 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca\") pod \"37c19139-3cec-45bf-9911-586887b31e85\" (UID: \"37c19139-3cec-45bf-9911-586887b31e85\") " Nov 01 15:08:46.379047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.378986 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca" pod "37c19139-3cec-45bf-9911-586887b31e85" (UID: "37c19139-3cec-45bf-9911-586887b31e85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:08:46.379361 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.379342 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37c19139-3cec-45bf-9911-586887b31e85-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:46.380875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.380847 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt" pod "37c19139-3cec-45bf-9911-586887b31e85" (UID: "37c19139-3cec-45bf-9911-586887b31e85"). InnerVolumeSpecName "kube-api-access-mvxgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:08:46.380875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.380856 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca" pod "37c19139-3cec-45bf-9911-586887b31e85" (UID: "37c19139-3cec-45bf-9911-586887b31e85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:08:46.481086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.481052 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37c19139-3cec-45bf-9911-586887b31e85-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:46.481270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:46.481090 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxgt\" (UniqueName: \"kubernetes.io/projected/37c19139-3cec-45bf-9911-586887b31e85-kube-api-access-mvxgt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:47.051109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:47.051077 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" Nov 01 15:08:47.051305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:47.051118 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp" event={"ID":"37c19139-3cec-45bf-9911-586887b31e85","Type":"ContainerDied","Data":"d665252e79fc7a49cf1f2cb4b96e4aba102ff21f16e4f7d5df233a615050c421"} Nov 01 15:08:47.051305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:47.051173 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d665252e79fc7a49cf1f2cb4b96e4aba102ff21f16e4f7d5df233a615050c421" Nov 01 15:08:50.642539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:50.642456 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:54.940410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.940371 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq"] Nov 01 15:08:54.941069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.941053 5473 cpu_manager.go:414] "containerMap: removing container" podUID="37c19139-3cec-45bf-9911-586887b31e85" containerName="revoke-keys" Nov 01 15:08:54.941128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.941071 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37c19139-3cec-45bf-9911-586887b31e85" containerName="revoke-keys" Nov 01 15:08:54.941186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.941177 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="37c19139-3cec-45bf-9911-586887b31e85" containerName="revoke-keys" Nov 01 15:08:54.945064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.945049 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:54.971887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.971858 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:54.972060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.971944 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:54.972060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.971993 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjsf\" (UniqueName: \"kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:54.997729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:54.997699 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq"] Nov 01 15:08:55.044559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.044526 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:08:55.044751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.044639 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:08:55.073586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.073557 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.073778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.073600 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjsf\" (UniqueName: \"kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.073778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.073732 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.074363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.074339 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.076083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.076060 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.131990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.131949 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjsf\" (UniqueName: \"kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf\") pod \"maas-api-revoke-sub-6bf9a36d-l2jbq\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.200582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.200498 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:55.258589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.258554 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:55.376803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.376768 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:55.627123 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:08:55.627094 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf323fb7a_8c38_4c15_bd5a_c6e21eacb5a4.slice/crio-a48b0cda9b54213fdf2e3703566c9bc57c40e46d20cbc4c92e63bbcb77480584 WatchSource:0}: Error finding container a48b0cda9b54213fdf2e3703566c9bc57c40e46d20cbc4c92e63bbcb77480584: Status 404 returned error can't find the container with id a48b0cda9b54213fdf2e3703566c9bc57c40e46d20cbc4c92e63bbcb77480584 Nov 01 15:08:55.629918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:55.629892 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq"] Nov 01 15:08:56.113508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:56.113474 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" event={"ID":"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4","Type":"ContainerStarted","Data":"68c72189d240088d87f63db8c38bb3615ecd0f5e4485cb46961fe6bf888af205"} Nov 01 15:08:56.114050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:56.113521 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" event={"ID":"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4","Type":"ContainerStarted","Data":"a48b0cda9b54213fdf2e3703566c9bc57c40e46d20cbc4c92e63bbcb77480584"} Nov 01 15:08:56.229319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:56.229204 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" podStartSLOduration=2.229188679 podStartE2EDuration="2.229188679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:08:56.227571645 +0000 UTC m=+3075.285723495" watchObservedRunningTime="2026-11-01 15:08:56.229188679 +0000 UTC m=+3075.287340528" Nov 01 15:08:57.121100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.121066 5473 generic.go:356] "Generic (PLEG): container finished" podID="f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" containerID="68c72189d240088d87f63db8c38bb3615ecd0f5e4485cb46961fe6bf888af205" exitCode=0 Nov 01 15:08:57.121595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.121149 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" event={"ID":"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4","Type":"ContainerDied","Data":"68c72189d240088d87f63db8c38bb3615ecd0f5e4485cb46961fe6bf888af205"} Nov 01 15:08:57.304773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.304730 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp"] Nov 01 15:08:57.308951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.308931 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.361691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.361661 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-db6f2e\"/\"openshift-service-ca.crt\"" Nov 01 15:08:57.361878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.361797 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-db6f2e\"/\"kube-root-ca.crt\"" Nov 01 15:08:57.361878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.361819 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-db6f2e\"/\"noten-model-db6f2e-kserve-self-signed-certs\"" Nov 01 15:08:57.361878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.361858 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-db6f2e\"/\"default-dockercfg-bdkvr\"" Nov 01 15:08:57.364343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.364320 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp"] Nov 01 15:08:57.399639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.399564 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.399787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.399724 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.399787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.399766 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.399883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.399866 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.399922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.399906 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczjc\" (UniqueName: \"kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.399985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.399960 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.437080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.437048 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:57.500809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.500780 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.500826 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.500878 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.500924 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczjc\" (UniqueName: \"kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.500968 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.501017 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.501278 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.501291 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.501482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.501465 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.503205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.503183 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.503692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.503674 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.557490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.557461 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczjc\" (UniqueName: \"kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc\") pod \"noten-model-db6f2e-kserve-6d5594657c-lzszp\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.621619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.621585 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:57.991670 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:08:57.991642 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd12f6fd4_7740_4fb0_b9fa_5c3aa45889c7.slice/crio-fd796c3c5944b6fdf75bb3d95ca9a9517e4fb7fe2ae8d899add96fa2479a2788 WatchSource:0}: Error finding container fd796c3c5944b6fdf75bb3d95ca9a9517e4fb7fe2ae8d899add96fa2479a2788: Status 404 returned error can't find the container with id fd796c3c5944b6fdf75bb3d95ca9a9517e4fb7fe2ae8d899add96fa2479a2788 Nov 01 15:08:57.994868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:57.994841 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp"] Nov 01 15:08:58.129694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.129634 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" event={"ID":"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7","Type":"ContainerStarted","Data":"64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a"} Nov 01 15:08:58.130118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.129709 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" event={"ID":"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7","Type":"ContainerStarted","Data":"fd796c3c5944b6fdf75bb3d95ca9a9517e4fb7fe2ae8d899add96fa2479a2788"} Nov 01 15:08:58.247836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.247727 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" podStartSLOduration=1.2477103760000001 podStartE2EDuration="1.247710376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:08:58.245951851 +0000 UTC m=+3077.304103701" watchObservedRunningTime="2026-11-01 15:08:58.247710376 +0000 UTC m=+3077.305862226" Nov 01 15:08:58.271254 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.271229 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:58.310525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.310492 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca\") pod \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " Nov 01 15:08:58.310768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.310752 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf\" (UniqueName: \"kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf\") pod \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " Nov 01 15:08:58.310849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.310790 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca\") pod \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\" (UID: \"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4\") " Nov 01 15:08:58.311181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.311155 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca" pod "f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" (UID: "f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:08:58.312597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.312571 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca" pod "f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" (UID: "f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:08:58.312699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.312677 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf" pod "f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" (UID: "f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4"). InnerVolumeSpecName "kube-api-access-xmjsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:08:58.411892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.411862 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjsf\" (UniqueName: \"kubernetes.io/projected/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-kube-api-access-xmjsf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:58.411892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.411889 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:58.412105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.411900 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:08:58.622056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.622021 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:08:58.872501 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:58.872424 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:08:59.138233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:59.138130 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" Nov 01 15:08:59.138233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:59.138168 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq" event={"ID":"f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4","Type":"ContainerDied","Data":"a48b0cda9b54213fdf2e3703566c9bc57c40e46d20cbc4c92e63bbcb77480584"} Nov 01 15:08:59.138683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:59.138242 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a48b0cda9b54213fdf2e3703566c9bc57c40e46d20cbc4c92e63bbcb77480584" Nov 01 15:08:59.306646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:08:59.306591 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:00.004113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:00.004087 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:01.108376 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:01.108342 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:09:04.386415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:04.386382 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:05.683042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.683007 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x"] Nov 01 15:09:05.683626 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.683610 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" containerName="revoke-keys" Nov 01 15:09:05.683669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.683628 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" containerName="revoke-keys" Nov 01 15:09:05.683738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.683729 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" containerName="revoke-keys" Nov 01 15:09:05.687384 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.687368 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.753836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.753804 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x"] Nov 01 15:09:05.800264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.800202 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:09:05.800442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.800275 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:09:05.800716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.800692 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.800770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.800743 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2g55\" (UniqueName: \"kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.800996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.800977 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.902491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.902456 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.902681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.902521 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.902681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.902577 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2g55\" (UniqueName: \"kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.903173 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.903154 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.904775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.904752 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:05.959492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:05.959425 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2g55\" (UniqueName: \"kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55\") pod \"maas-api-revoke-sub-32ccbb09-rzx7x\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:06.000616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:06.000582 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:06.376871 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:09:06.376840 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19fabe0_0f41_4d1b_98be_544aed6c41c4.slice/crio-760387eeb60204408eb403a1b2e95ec4ccbfcb524b52c61e360815d990eaa7c7 WatchSource:0}: Error finding container 760387eeb60204408eb403a1b2e95ec4ccbfcb524b52c61e360815d990eaa7c7: Status 404 returned error can't find the container with id 760387eeb60204408eb403a1b2e95ec4ccbfcb524b52c61e360815d990eaa7c7 Nov 01 15:09:06.380793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:06.380771 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x"] Nov 01 15:09:07.192263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:07.192223 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" event={"ID":"c19fabe0-0f41-4d1b-98be-544aed6c41c4","Type":"ContainerStarted","Data":"0f191dc3e73c682628ef5a9447b66ea5761f2432893d3b11c04a49cd067d6c6b"} Nov 01 15:09:07.192670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:07.192279 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" event={"ID":"c19fabe0-0f41-4d1b-98be-544aed6c41c4","Type":"ContainerStarted","Data":"760387eeb60204408eb403a1b2e95ec4ccbfcb524b52c61e360815d990eaa7c7"} Nov 01 15:09:07.305833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:07.305770 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" podStartSLOduration=2.3057574 podStartE2EDuration="2.3057574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:09:07.304741811 +0000 UTC m=+3086.362893665" watchObservedRunningTime="2026-11-01 15:09:07.3057574 +0000 UTC m=+3086.363909250" Nov 01 15:09:07.622163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:07.622133 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:09:07.634460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:07.634437 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:09:08.200474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:08.200443 5473 generic.go:356] "Generic (PLEG): container finished" podID="c19fabe0-0f41-4d1b-98be-544aed6c41c4" containerID="0f191dc3e73c682628ef5a9447b66ea5761f2432893d3b11c04a49cd067d6c6b" exitCode=0 Nov 01 15:09:08.200895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:08.200535 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" event={"ID":"c19fabe0-0f41-4d1b-98be-544aed6c41c4","Type":"ContainerDied","Data":"0f191dc3e73c682628ef5a9447b66ea5761f2432893d3b11c04a49cd067d6c6b"} Nov 01 15:09:08.212010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:08.211988 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:09:08.894665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:08.894635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:09.353032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.353005 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:09.450915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.450885 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55\" (UniqueName: \"kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55\") pod \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " Nov 01 15:09:09.451080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.450924 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca\") pod \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " Nov 01 15:09:09.451124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.451077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca\") pod \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\" (UID: \"c19fabe0-0f41-4d1b-98be-544aed6c41c4\") " Nov 01 15:09:09.451456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.451428 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca" pod "c19fabe0-0f41-4d1b-98be-544aed6c41c4" (UID: "c19fabe0-0f41-4d1b-98be-544aed6c41c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:09:09.451671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.451652 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:09.452979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.452951 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55" pod "c19fabe0-0f41-4d1b-98be-544aed6c41c4" (UID: "c19fabe0-0f41-4d1b-98be-544aed6c41c4"). InnerVolumeSpecName "kube-api-access-b2g55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:09:09.453072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.453021 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca" pod "c19fabe0-0f41-4d1b-98be-544aed6c41c4" (UID: "c19fabe0-0f41-4d1b-98be-544aed6c41c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:09:09.553257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.553206 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2g55\" (UniqueName: \"kubernetes.io/projected/c19fabe0-0f41-4d1b-98be-544aed6c41c4-kube-api-access-b2g55\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:09.553257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:09.553259 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c19fabe0-0f41-4d1b-98be-544aed6c41c4-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:10.218434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:10.218406 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" Nov 01 15:09:10.218616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:10.218440 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x" event={"ID":"c19fabe0-0f41-4d1b-98be-544aed6c41c4","Type":"ContainerDied","Data":"760387eeb60204408eb403a1b2e95ec4ccbfcb524b52c61e360815d990eaa7c7"} Nov 01 15:09:10.218616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:10.218485 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="760387eeb60204408eb403a1b2e95ec4ccbfcb524b52c61e360815d990eaa7c7" Nov 01 15:09:13.300685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:13.300651 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:17.697167 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:17.697132 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:22.132591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:22.132547 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:26.499871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:26.499834 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:31.013705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:31.013660 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:31.107792 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:31.107754 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:09:35.432578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:35.432534 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:35.513300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:35.513267 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d"] Nov 01 15:09:35.513572 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:35.513543 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" podUID="a1ee1935-935d-414a-a848-0c0ce0f607d4" containerName="istio-proxy" containerID="cri-o://8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c" gracePeriod=30 Nov 01 15:09:35.623251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:35.623196 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:35.863472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:35.863437 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:36.038698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:36.038666 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:37.984781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:37.984742 5473 patch_prober.go:29] interesting pod/e2e-noten-db6f2e-openshift-default-f685796f9-s776d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:09:37.985210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:37.984839 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" podUID="a1ee1935-935d-414a-a848-0c0ce0f607d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:09:40.782670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.782612 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:09:40.837365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837332 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837426 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837473 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837523 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837560 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837562 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:40.837667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837587 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837707 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:40.837788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837755 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837805 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:40.837899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837816 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:40.837899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837842 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.837899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.837890 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo\") pod \"a1ee1935-935d-414a-a848-0c0ce0f607d4\" (UID: \"a1ee1935-935d-414a-a848-0c0ce0f607d4\") " Nov 01 15:09:40.838159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.838132 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:09:40.838346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.838327 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.838428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.838351 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.838428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.838365 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.838428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.838378 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ee1935-935d-414a-a848-0c0ce0f607d4-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.838428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.838391 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.839829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.839802 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:40.839991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.839954 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "kube-api-access-vq5wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:09:40.840284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.840264 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:09:40.840358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.840311 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token" pod "a1ee1935-935d-414a-a848-0c0ce0f607d4" (UID: "a1ee1935-935d-414a-a848-0c0ce0f607d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:09:40.917345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.917277 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp"] Nov 01 15:09:40.940028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.939997 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5wz\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-kube-api-access-vq5wz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.940028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.940025 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.940028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.940035 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:40.940280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:40.940045 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ee1935-935d-414a-a848-0c0ce0f607d4-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:41.446132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.446048 5473 generic.go:356] "Generic (PLEG): container finished" podID="a1ee1935-935d-414a-a848-0c0ce0f607d4" containerID="8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c" exitCode=0 Nov 01 15:09:41.446132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.446092 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" Nov 01 15:09:41.446379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.446127 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" event={"ID":"a1ee1935-935d-414a-a848-0c0ce0f607d4","Type":"ContainerDied","Data":"8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c"} Nov 01 15:09:41.446379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.446181 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d" event={"ID":"a1ee1935-935d-414a-a848-0c0ce0f607d4","Type":"ContainerDied","Data":"ed9f52813b933105e484afa83f20884df9e5a4e2da6fd69d758de590dbf9b96f"} Nov 01 15:09:41.446379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.446207 5473 scope.go:122] "RemoveContainer" containerID="8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c" Nov 01 15:09:41.456703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.456681 5473 scope.go:122] "RemoveContainer" containerID="8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c" Nov 01 15:09:41.457017 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:41.456997 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c\": container with ID starting with 8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c not found: ID does not exist" containerID="8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c" Nov 01 15:09:41.457106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.457027 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c"} err="failed to get container status \"8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c\": rpc error: code = NotFound desc = could not find container \"8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c\": container with ID starting with 8298102b99fde46aa92009afbbadbd45ad63a17345506b965d65d314cf963c7c not found: ID does not exist" Nov 01 15:09:41.613647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.613609 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d"] Nov 01 15:09:41.616837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.616813 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-db6f2e-openshift-default-f685796f9-s776d"] Nov 01 15:09:41.943950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:41.943920 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:42.108574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:42.108541 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:43.082586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:43.082552 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ee1935-935d-414a-a848-0c0ce0f607d4" path="/var/lib/kubelet/pods/a1ee1935-935d-414a-a848-0c0ce0f607d4/volumes" Nov 01 15:09:44.409103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:44.409072 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:48.902400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:48.902363 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:51.212857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.212818 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7"] Nov 01 15:09:51.261096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.261058 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:51.467236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.467148 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7381-df5bea"] Nov 01 15:09:51.468124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.468103 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c19fabe0-0f41-4d1b-98be-544aed6c41c4" containerName="revoke-keys" Nov 01 15:09:51.468185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.468128 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c19fabe0-0f41-4d1b-98be-544aed6c41c4" containerName="revoke-keys" Nov 01 15:09:51.468185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.468143 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a1ee1935-935d-414a-a848-0c0ce0f607d4" containerName="istio-proxy" Nov 01 15:09:51.468185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.468152 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ee1935-935d-414a-a848-0c0ce0f607d4" containerName="istio-proxy" Nov 01 15:09:51.468324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.468305 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c19fabe0-0f41-4d1b-98be-544aed6c41c4" containerName="revoke-keys" Nov 01 15:09:51.468324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.468320 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ee1935-935d-414a-a848-0c0ce0f607d4" containerName="istio-proxy" Nov 01 15:09:51.474206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.474184 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:51.532814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.532784 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\"" Nov 01 15:09:51.534772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.534743 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-df5bea"] Nov 01 15:09:51.558447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.558421 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtbf\" (UniqueName: \"kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf\") pod \"test-curl-7381-df5bea\" (UID: \"28dc0af1-47ba-4c35-80cc-72f98db44ad4\") " pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:51.659569 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.659486 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtbf\" (UniqueName: \"kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf\") pod \"test-curl-7381-df5bea\" (UID: \"28dc0af1-47ba-4c35-80cc-72f98db44ad4\") " pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:51.718413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.718341 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtbf\" (UniqueName: \"kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf\") pod \"test-curl-7381-df5bea\" (UID: \"28dc0af1-47ba-4c35-80cc-72f98db44ad4\") " pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:51.789071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:51.789044 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:52.161026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:52.161001 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:09:52.162812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:52.162735 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7381-df5bea"] Nov 01 15:09:52.253568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:52.253529 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:52.523812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:52.523772 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-df5bea" event={"ID":"28dc0af1-47ba-4c35-80cc-72f98db44ad4","Type":"ContainerStarted","Data":"f13967761c7b2ae20a448e9caf5eae1b5b03471e1492ede2c184323a8aa0d9eb"} Nov 01 15:09:53.227388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:53.227347 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:53.411271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:53.411208 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:54.781588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.781555 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5"] Nov 01 15:09:54.786936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.786918 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.840767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.840734 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5"] Nov 01 15:09:54.885914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.885874 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"openshift-service-ca.crt\"" Nov 01 15:09:54.885914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.885874 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"kube-root-ca.crt\"" Nov 01 15:09:54.885914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.885906 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"auth-test-model-1ac11c-kserve-self-signed-certs\"" Nov 01 15:09:54.889430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.889412 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"default-dockercfg-tzv99\"" Nov 01 15:09:54.899897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.899871 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.900012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.899911 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.900064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.900052 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.900102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.900086 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.900158 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.900140 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47jpx\" (UniqueName: \"kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.900274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.900258 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:54.913637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:54.913613 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:55.001777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.001748 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.001988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.001787 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.001988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.001873 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.001988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.001905 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.001988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.001945 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47jpx\" (UniqueName: \"kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.001988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.001987 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.002284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.002204 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.002358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.002290 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.002422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.002368 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.004152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.004132 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.004449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.004432 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.059368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.059306 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47jpx\" (UniqueName: \"kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx\") pod \"auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.099834 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.099808 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:55.474322 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:09:55.474286 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83ca903f_828b_47b9_b2ac_7b8cc40332b1.slice/crio-59d6d767c2d92ba3ffe6031dd517ade7a5cdcb1fe3b6e0a7fa374dc410d4759b WatchSource:0}: Error finding container 59d6d767c2d92ba3ffe6031dd517ade7a5cdcb1fe3b6e0a7fa374dc410d4759b: Status 404 returned error can't find the container with id 59d6d767c2d92ba3ffe6031dd517ade7a5cdcb1fe3b6e0a7fa374dc410d4759b Nov 01 15:09:55.476842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.476815 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5"] Nov 01 15:09:55.545691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.545650 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" event={"ID":"83ca903f-828b-47b9-b2ac-7b8cc40332b1","Type":"ContainerStarted","Data":"59d6d767c2d92ba3ffe6031dd517ade7a5cdcb1fe3b6e0a7fa374dc410d4759b"} Nov 01 15:09:55.938866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:55.938829 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" podUID="d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" containerName="main" containerID="cri-o://64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a" gracePeriod=45 Nov 01 15:09:56.152348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.152319 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-ceb31a"] Nov 01 15:09:56.156577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.156557 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:09:56.204145 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.204080 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:09:56.204349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.204331 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-ceb31a"] Nov 01 15:09:56.216484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.216458 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qz5f\" (UniqueName: \"kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f\") pod \"test-curl-7384-ceb31a\" (UID: \"7850b28e-5189-4f27-8207-6d2e15c57696\") " pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:09:56.317630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317603 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache\") pod \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " Nov 01 15:09:56.317766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317689 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir\") pod \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " Nov 01 15:09:56.317814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317770 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc\" (UniqueName: \"kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc\") pod \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " Nov 01 15:09:56.317814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317810 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm\") pod \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " Nov 01 15:09:56.317932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317849 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache" pod "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" (UID: "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:56.317932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317880 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home\") pod \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " Nov 01 15:09:56.317932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317899 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs\") pod \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\" (UID: \"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7\") " Nov 01 15:09:56.318105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.317946 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir" pod "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" (UID: "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:56.318105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.318022 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qz5f\" (UniqueName: \"kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f\") pod \"test-curl-7384-ceb31a\" (UID: \"7850b28e-5189-4f27-8207-6d2e15c57696\") " pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:09:56.318105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.318075 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home" pod "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" (UID: "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:56.318336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.318315 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:56.318388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.318342 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:56.318388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.318358 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:56.320172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.320151 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs" pod "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" (UID: "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:09:56.320444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.320428 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm" pod "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" (UID: "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:09:56.320511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.320489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc" pod "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" (UID: "d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7"). InnerVolumeSpecName "kube-api-access-vczjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:09:56.385245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.385198 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qz5f\" (UniqueName: \"kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f\") pod \"test-curl-7384-ceb31a\" (UID: \"7850b28e-5189-4f27-8207-6d2e15c57696\") " pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:09:56.419802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.419768 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vczjc\" (UniqueName: \"kubernetes.io/projected/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-kube-api-access-vczjc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:56.419920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.419807 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:56.419920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.419822 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:09:56.469122 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.469060 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:09:56.554708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.554681 5473 generic.go:356] "Generic (PLEG): container finished" podID="d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" containerID="64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a" exitCode=0 Nov 01 15:09:56.554850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.554741 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" Nov 01 15:09:56.554850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.554768 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" event={"ID":"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7","Type":"ContainerDied","Data":"64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a"} Nov 01 15:09:56.554850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.554811 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp" event={"ID":"d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7","Type":"ContainerDied","Data":"fd796c3c5944b6fdf75bb3d95ca9a9517e4fb7fe2ae8d899add96fa2479a2788"} Nov 01 15:09:56.554850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.554832 5473 scope.go:122] "RemoveContainer" containerID="64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a" Nov 01 15:09:56.556969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.556941 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" event={"ID":"83ca903f-828b-47b9-b2ac-7b8cc40332b1","Type":"ContainerStarted","Data":"ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747"} Nov 01 15:09:56.566245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.566204 5473 scope.go:122] "RemoveContainer" containerID="64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a" Nov 01 15:09:56.566548 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:56.566526 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a\": container with ID starting with 64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a not found: ID does not exist" containerID="64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a" Nov 01 15:09:56.566609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.566560 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a"} err="failed to get container status \"64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a\": rpc error: code = NotFound desc = could not find container \"64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a\": container with ID starting with 64c7ff7f971127c9637bae786197f61d6967637812ba5e220f041b7c14d0034a not found: ID does not exist" Nov 01 15:09:56.725267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.725176 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" podStartSLOduration=2.72516199 podStartE2EDuration="2.72516199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:09:56.723201281 +0000 UTC m=+3135.781353132" watchObservedRunningTime="2026-11-01 15:09:56.72516199 +0000 UTC m=+3135.783313857" Nov 01 15:09:56.838928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.838895 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp"] Nov 01 15:09:56.844228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.844186 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-ceb31a"] Nov 01 15:09:56.848952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:56.848913 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-db6f2e/noten-model-db6f2e-kserve-6d5594657c-lzszp"] Nov 01 15:09:57.082251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.082201 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" path="/var/lib/kubelet/pods/d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7/volumes" Nov 01 15:09:57.100684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.100659 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:09:57.565068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.565031 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7381-df5bea" event={"ID":"28dc0af1-47ba-4c35-80cc-72f98db44ad4","Type":"ContainerStarted","Data":"b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1"} Nov 01 15:09:57.566665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.566635 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-ceb31a" event={"ID":"7850b28e-5189-4f27-8207-6d2e15c57696","Type":"ContainerStarted","Data":"c31b6a8fcd530300f2b986c396b10781724fe14cb40a409305e13bb61324a2e3"} Nov 01 15:09:57.566814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.566677 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-ceb31a" event={"ID":"7850b28e-5189-4f27-8207-6d2e15c57696","Type":"ContainerStarted","Data":"de29f7a8a9056bd3469943343fd23825c814aa5d5e2fcb946a4ad346369ce55a"} Nov 01 15:09:57.638440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.638418 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:57.680133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.680079 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7381-df5bea" podStartSLOduration=2.211273946 podStartE2EDuration="6.680065501s" totalImagesPullingTime="4.468791555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:09:57.677603529 +0000 UTC m=+3136.735755378" watchObservedRunningTime="2026-11-01 15:09:57.680065501 +0000 UTC m=+3136.738217350" Nov 01 15:09:57.793440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:57.793391 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-ceb31a" podStartSLOduration=1.793377504 podStartE2EDuration="1.793377504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:09:57.790361508 +0000 UTC m=+3136.848513358" watchObservedRunningTime="2026-11-01 15:09:57.793377504 +0000 UTC m=+3136.851529354" Nov 01 15:09:59.182955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.182840 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-ceb31a"] Nov 01 15:09:59.188080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.188051 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-ceb31a"] Nov 01 15:09:59.259309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.259277 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7381-df5bea"] Nov 01 15:09:59.262349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.262325 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7381-df5bea"] Nov 01 15:09:59.323736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.323708 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-ebdc8b"] Nov 01 15:09:59.324276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324262 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" containerName="main" Nov 01 15:09:59.324327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324277 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" containerName="main" Nov 01 15:09:59.324327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324309 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" containerName="test-curl-7384-ceb31a" Nov 01 15:09:59.324327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324315 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" containerName="test-curl-7384-ceb31a" Nov 01 15:09:59.324430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324332 5473 cpu_manager.go:414] "containerMap: removing container" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" containerName="test-curl-7381-df5bea" Nov 01 15:09:59.324430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324337 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" containerName="test-curl-7381-df5bea" Nov 01 15:09:59.324430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324419 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" containerName="test-curl-7384-ceb31a" Nov 01 15:09:59.324527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324431 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" containerName="test-curl-7381-df5bea" Nov 01 15:09:59.324527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.324442 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12f6fd4-7740-4fb0-b9fa-5c3aa45889c7" containerName="main" Nov 01 15:09:59.328091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.328075 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:09:59.378730 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.378703 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-ebdc8b"] Nov 01 15:09:59.455562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.455502 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gn2j\" (UniqueName: \"kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j\") pod \"test-curl-7384-ebdc8b\" (UID: \"4a62a406-076f-45cc-884c-05eb720d2e8d\") " pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:09:59.556467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.556437 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gn2j\" (UniqueName: \"kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j\") pod \"test-curl-7384-ebdc8b\" (UID: \"4a62a406-076f-45cc-884c-05eb720d2e8d\") " pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:09:59.582615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.582586 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-ceb31a" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" containerName="test-curl-7384-ceb31a" containerID="cri-o://c31b6a8fcd530300f2b986c396b10781724fe14cb40a409305e13bb61324a2e3" gracePeriod=2 Nov 01 15:09:59.582795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.582646 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7381-df5bea" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" containerName="test-curl-7381-df5bea" containerID="cri-o://b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1" gracePeriod=2 Nov 01 15:09:59.612892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.612870 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gn2j\" (UniqueName: \"kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j\") pod \"test-curl-7384-ebdc8b\" (UID: \"4a62a406-076f-45cc-884c-05eb720d2e8d\") " pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:09:59.630859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.630823 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:09:59.633733 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:59.633710 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-df5bea\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:59.640688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:09:59.640668 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:09:59.684805 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:59.684771 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ceb31a\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:09:59.736144 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:59.736066 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-df5bea\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:09:59.787121 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:09:59.787084 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ceb31a\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:10:00.004322 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:00.004288 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a62a406_076f_45cc_884c_05eb720d2e8d.slice/crio-53bb02009b33c3c2ee0867bc70c3aae1ddfa09826453f790e0f6431cc51e24ab WatchSource:0}: Error finding container 53bb02009b33c3c2ee0867bc70c3aae1ddfa09826453f790e0f6431cc51e24ab: Status 404 returned error can't find the container with id 53bb02009b33c3c2ee0867bc70c3aae1ddfa09826453f790e0f6431cc51e24ab Nov 01 15:10:00.010862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:00.010833 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-ebdc8b"] Nov 01 15:10:00.593600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:00.593558 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-ebdc8b" event={"ID":"4a62a406-076f-45cc-884c-05eb720d2e8d","Type":"ContainerStarted","Data":"7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711"} Nov 01 15:10:00.593600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:00.593603 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-ebdc8b" event={"ID":"4a62a406-076f-45cc-884c-05eb720d2e8d","Type":"ContainerStarted","Data":"53bb02009b33c3c2ee0867bc70c3aae1ddfa09826453f790e0f6431cc51e24ab"} Nov 01 15:10:00.644968 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:00.644933 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-df5bea\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:10:00.696064 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:00.696023 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ceb31a\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:10:00.810459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:00.810407 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-ebdc8b" podStartSLOduration=1.8103930190000002 podStartE2EDuration="1.810393019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:00.807733363 +0000 UTC m=+3139.865885213" watchObservedRunningTime="2026-11-01 15:10:00.810393019 +0000 UTC m=+3139.868544868" Nov 01 15:10:01.002793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.002721 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:01.108576 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:01.108543 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:10:01.132104 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:01.132070 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-df5bea\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:10:01.187694 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:01.187665 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ceb31a\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:10:01.469827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.469796 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx"] Nov 01 15:10:01.473936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.473916 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.526410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.526365 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0f1e68-openshift-default-dockercfg-pq647\"" Nov 01 15:10:01.578144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578110 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578354 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578412 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bcjh\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578509 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578585 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578562 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578678 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578721 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578753 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.578896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.578859 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.582088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.582068 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx"] Nov 01 15:10:01.584009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.583981 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:01.604262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.604241 5473 generic.go:356] "Generic (PLEG): container finished" podID="7850b28e-5189-4f27-8207-6d2e15c57696" containerID="c31b6a8fcd530300f2b986c396b10781724fe14cb40a409305e13bb61324a2e3" exitCode=137 Nov 01 15:10:01.680979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.680945 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681030 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681190 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681246 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bcjh\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681294 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681338 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681452 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681557 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.681635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.681596 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.682369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.682346 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.682672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.682649 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.682855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.682833 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.683257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.683233 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.683415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.683384 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.684909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.684883 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.686083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.686061 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.750686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.747169 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bcjh\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.756686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.756652 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token\") pod \"e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.790341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.790275 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:01.794945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.794904 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:01.821179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.821153 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:10:01.856251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.856210 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:10:01.984209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.984179 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf\" (UniqueName: \"kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf\") pod \"28dc0af1-47ba-4c35-80cc-72f98db44ad4\" (UID: \"28dc0af1-47ba-4c35-80cc-72f98db44ad4\") " Nov 01 15:10:01.984406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.984348 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f\" (UniqueName: \"kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f\") pod \"7850b28e-5189-4f27-8207-6d2e15c57696\" (UID: \"7850b28e-5189-4f27-8207-6d2e15c57696\") " Nov 01 15:10:01.986383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.986355 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf" pod "28dc0af1-47ba-4c35-80cc-72f98db44ad4" (UID: "28dc0af1-47ba-4c35-80cc-72f98db44ad4"). InnerVolumeSpecName "kube-api-access-rxtbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:01.986509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:01.986437 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f" pod "7850b28e-5189-4f27-8207-6d2e15c57696" (UID: "7850b28e-5189-4f27-8207-6d2e15c57696"). InnerVolumeSpecName "kube-api-access-9qz5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:02.071427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.071399 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:02.086973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.086937 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qz5f\" (UniqueName: \"kubernetes.io/projected/7850b28e-5189-4f27-8207-6d2e15c57696-kube-api-access-9qz5f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:02.086973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.086976 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxtbf\" (UniqueName: \"kubernetes.io/projected/28dc0af1-47ba-4c35-80cc-72f98db44ad4-kube-api-access-rxtbf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:02.217124 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:02.217096 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b49dc5d_d83c_4bf9_9f57_ef5d6cae298a.slice/crio-9790a69463cf583955c902ecec1c7e1967cfd30f82c47603471a8f7ad1ad9fdb WatchSource:0}: Error finding container 9790a69463cf583955c902ecec1c7e1967cfd30f82c47603471a8f7ad1ad9fdb: Status 404 returned error can't find the container with id 9790a69463cf583955c902ecec1c7e1967cfd30f82c47603471a8f7ad1ad9fdb Nov 01 15:10:02.220431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.220390 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:10:02.220533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.220431 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx"] Nov 01 15:10:02.220533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.220476 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:10:02.220533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.220514 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:10:02.296468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.296445 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-ebdc8b"] Nov 01 15:10:02.301477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.301454 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-ebdc8b"] Nov 01 15:10:02.432553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.432481 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-e7ce08"] Nov 01 15:10:02.433087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.433074 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" containerName="test-curl-7384-ebdc8b" Nov 01 15:10:02.433134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.433088 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" containerName="test-curl-7384-ebdc8b" Nov 01 15:10:02.433189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.433180 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" containerName="test-curl-7384-ebdc8b" Nov 01 15:10:02.436557 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.436541 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:02.486128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.486096 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-e7ce08"] Nov 01 15:10:02.593634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.593601 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbj74\" (UniqueName: \"kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74\") pod \"test-curl-7384-e7ce08\" (UID: \"36a4a488-2a74-4219-949a-2b13a2590f1a\") " pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:02.615052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.615013 5473 generic.go:356] "Generic (PLEG): container finished" podID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" containerID="b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1" exitCode=137 Nov 01 15:10:02.615489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.615115 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:10:02.615489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.615153 5473 scope.go:122] "RemoveContainer" containerID="b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1" Nov 01 15:10:02.616731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.616710 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:10:02.618946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.618914 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" event={"ID":"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a","Type":"ContainerStarted","Data":"9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339"} Nov 01 15:10:02.619067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.619000 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-ebdc8b" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" containerName="test-curl-7384-ebdc8b" containerID="cri-o://7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711" gracePeriod=2 Nov 01 15:10:02.619133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.619077 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" event={"ID":"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a","Type":"ContainerStarted","Data":"9790a69463cf583955c902ecec1c7e1967cfd30f82c47603471a8f7ad1ad9fdb"} Nov 01 15:10:02.627833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.627809 5473 scope.go:122] "RemoveContainer" containerID="b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1" Nov 01 15:10:02.628170 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:02.628145 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1\": container with ID starting with b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1 not found: ID does not exist" containerID="b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1" Nov 01 15:10:02.628259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.628197 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1"} err="failed to get container status \"b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1\": rpc error: code = NotFound desc = could not find container \"b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1\": container with ID starting with b2e90dc28761e3b752c218b316c2209c1c79dd9822af058a52e2311e562706d1 not found: ID does not exist" Nov 01 15:10:02.628259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.628242 5473 scope.go:122] "RemoveContainer" containerID="c31b6a8fcd530300f2b986c396b10781724fe14cb40a409305e13bb61324a2e3" Nov 01 15:10:02.670238 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:02.670197 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ebdc8b\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:10:02.694858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.694777 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbj74\" (UniqueName: \"kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74\") pod \"test-curl-7384-e7ce08\" (UID: \"36a4a488-2a74-4219-949a-2b13a2590f1a\") " pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:02.762163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.762137 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbj74\" (UniqueName: \"kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74\") pod \"test-curl-7384-e7ce08\" (UID: \"36a4a488-2a74-4219-949a-2b13a2590f1a\") " pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:02.790573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.790544 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:02.802903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:02.802856 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podStartSLOduration=1.802841415 podStartE2EDuration="1.802841415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:02.800812822 +0000 UTC m=+3141.858964683" watchObservedRunningTime="2026-11-01 15:10:02.802841415 +0000 UTC m=+3141.860993327" Nov 01 15:10:02.852249 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:02.852176 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ebdc8b\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:10:02.903189 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:02.903160 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7381-df5bea\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" pod="opendatahub/test-curl-7381-df5bea" Nov 01 15:10:02.956917 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:02.956813 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-ceb31a\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" pod="opendatahub/test-curl-7384-ceb31a" Nov 01 15:10:03.048987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.048954 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:03.085176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.085141 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dc0af1-47ba-4c35-80cc-72f98db44ad4" path="/var/lib/kubelet/pods/28dc0af1-47ba-4c35-80cc-72f98db44ad4/volumes" Nov 01 15:10:03.085584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.085568 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7850b28e-5189-4f27-8207-6d2e15c57696" path="/var/lib/kubelet/pods/7850b28e-5189-4f27-8207-6d2e15c57696/volumes" Nov 01 15:10:03.418086 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:03.418056 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36a4a488_2a74_4219_949a_2b13a2590f1a.slice/crio-5fd195735b322eda53e68b905f34590761a3cd2184303741502963e1fda4b7b0 WatchSource:0}: Error finding container 5fd195735b322eda53e68b905f34590761a3cd2184303741502963e1fda4b7b0: Status 404 returned error can't find the container with id 5fd195735b322eda53e68b905f34590761a3cd2184303741502963e1fda4b7b0 Nov 01 15:10:03.423255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.423132 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-e7ce08"] Nov 01 15:10:03.627205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.627168 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-e7ce08" event={"ID":"36a4a488-2a74-4219-949a-2b13a2590f1a","Type":"ContainerStarted","Data":"4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457"} Nov 01 15:10:03.627614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.627248 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-e7ce08" event={"ID":"36a4a488-2a74-4219-949a-2b13a2590f1a","Type":"ContainerStarted","Data":"5fd195735b322eda53e68b905f34590761a3cd2184303741502963e1fda4b7b0"} Nov 01 15:10:03.744249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.744125 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-e7ce08" podStartSLOduration=1.744113312 podStartE2EDuration="1.744113312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:03.742519423 +0000 UTC m=+3142.800671270" watchObservedRunningTime="2026-11-01 15:10:03.744113312 +0000 UTC m=+3142.802265161" Nov 01 15:10:03.796995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:03.796969 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:04.636759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:04.636732 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:04.637525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:04.637503 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:10:04.860681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:04.860650 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:10:05.023205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.023175 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j\" (UniqueName: \"kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j\") pod \"4a62a406-076f-45cc-884c-05eb720d2e8d\" (UID: \"4a62a406-076f-45cc-884c-05eb720d2e8d\") " Nov 01 15:10:05.025151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.025124 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j" pod "4a62a406-076f-45cc-884c-05eb720d2e8d" (UID: "4a62a406-076f-45cc-884c-05eb720d2e8d"). InnerVolumeSpecName "kube-api-access-9gn2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:05.085649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.085610 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a62a406-076f-45cc-884c-05eb720d2e8d" path="/var/lib/kubelet/pods/4a62a406-076f-45cc-884c-05eb720d2e8d/volumes" Nov 01 15:10:05.086315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.086292 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-e7ce08"] Nov 01 15:10:05.090016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.089991 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-e7ce08"] Nov 01 15:10:05.100511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.100492 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:10:05.113904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.113884 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:10:05.125182 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.125156 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gn2j\" (UniqueName: \"kubernetes.io/projected/4a62a406-076f-45cc-884c-05eb720d2e8d-kube-api-access-9gn2j\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:05.263223 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.263175 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n"] Nov 01 15:10:05.263747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.263733 5473 cpu_manager.go:414] "containerMap: removing container" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" containerName="test-curl-7384-e7ce08" Nov 01 15:10:05.263800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.263748 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" containerName="test-curl-7384-e7ce08" Nov 01 15:10:05.263861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.263852 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" containerName="test-curl-7384-e7ce08" Nov 01 15:10:05.267690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.267672 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.324687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.324660 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n"] Nov 01 15:10:05.383939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.383905 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:10:05.384134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.384115 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:10:05.428907 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.428878 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.429061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.428936 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.429061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.429033 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn77w\" (UniqueName: \"kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.530729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.530649 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.530879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.530735 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.530879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.530789 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn77w\" (UniqueName: \"kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.531329 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.531305 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.533076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.533052 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.588610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.588583 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn77w\" (UniqueName: \"kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w\") pod \"maas-api-revoke-sub-a1a45376-4958n\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:05.644098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.644072 5473 generic.go:356] "Generic (PLEG): container finished" podID="4a62a406-076f-45cc-884c-05eb720d2e8d" containerID="7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711" exitCode=137 Nov 01 15:10:05.644484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.644112 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-ebdc8b" Nov 01 15:10:05.644484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.644161 5473 scope.go:122] "RemoveContainer" containerID="7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711" Nov 01 15:10:05.644484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.644432 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-e7ce08" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" containerName="test-curl-7384-e7ce08" containerID="cri-o://4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457" gracePeriod=2 Nov 01 15:10:05.656933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.656917 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:10:05.658500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.658483 5473 scope.go:122] "RemoveContainer" containerID="7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711" Nov 01 15:10:05.658761 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:05.658742 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711\": container with ID starting with 7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711 not found: ID does not exist" containerID="7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711" Nov 01 15:10:05.658817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.658768 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711"} err="failed to get container status \"7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711\": rpc error: code = NotFound desc = could not find container \"7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711\": container with ID starting with 7d51a158cd0590415bb32723e7bfcac985ceed6dd96d5da3358bad0658471711 not found: ID does not exist" Nov 01 15:10:05.695643 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:05.695617 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-e7ce08\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:05.747029 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:05.747001 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-e7ce08\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:05.882204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:05.882134 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:06.234810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.234732 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" podUID="0cd91561-4810-47d6-a24c-a6ee5a97d9f8" containerName="main" containerID="cri-o://9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8" gracePeriod=45 Nov 01 15:10:06.254465 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:06.254439 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9843443_a8bf_4e3f_8cb5_da078eab35da.slice/crio-500cb72728626d8d9f5d98dec6bb3337ddb80fccba279bcd320edea49872dcc8 WatchSource:0}: Error finding container 500cb72728626d8d9f5d98dec6bb3337ddb80fccba279bcd320edea49872dcc8: Status 404 returned error can't find the container with id 500cb72728626d8d9f5d98dec6bb3337ddb80fccba279bcd320edea49872dcc8 Nov 01 15:10:06.257205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.257180 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n"] Nov 01 15:10:06.515782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.515756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:06.551100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.551079 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:10:06.602539 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:06.602506 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-e7ce08\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:06.645104 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645076 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs\") pod \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " Nov 01 15:10:06.645543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645134 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache\") pod \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " Nov 01 15:10:06.645543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645195 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir\") pod \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " Nov 01 15:10:06.645543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645258 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm\") pod \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " Nov 01 15:10:06.645543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645428 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache" pod "0cd91561-4810-47d6-a24c-a6ee5a97d9f8" (UID: "0cd91561-4810-47d6-a24c-a6ee5a97d9f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:10:06.645543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645471 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7\" (UniqueName: \"kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7\") pod \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " Nov 01 15:10:06.645543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645506 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir" pod "0cd91561-4810-47d6-a24c-a6ee5a97d9f8" (UID: "0cd91561-4810-47d6-a24c-a6ee5a97d9f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:10:06.645838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.645817 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home\") pod \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\" (UID: \"0cd91561-4810-47d6-a24c-a6ee5a97d9f8\") " Nov 01 15:10:06.646111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.646074 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home" pod "0cd91561-4810-47d6-a24c-a6ee5a97d9f8" (UID: "0cd91561-4810-47d6-a24c-a6ee5a97d9f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:10:06.646461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.646443 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:06.646580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.646465 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:06.646580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.646479 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:06.647783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.647759 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm" pod "0cd91561-4810-47d6-a24c-a6ee5a97d9f8" (UID: "0cd91561-4810-47d6-a24c-a6ee5a97d9f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:10:06.647986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.647914 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs" pod "0cd91561-4810-47d6-a24c-a6ee5a97d9f8" (UID: "0cd91561-4810-47d6-a24c-a6ee5a97d9f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:10:06.648041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.648018 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7" pod "0cd91561-4810-47d6-a24c-a6ee5a97d9f8" (UID: "0cd91561-4810-47d6-a24c-a6ee5a97d9f8"). InnerVolumeSpecName "kube-api-access-tgnk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:06.653604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.653575 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" event={"ID":"f9843443-a8bf-4e3f-8cb5-da078eab35da","Type":"ContainerStarted","Data":"500cb72728626d8d9f5d98dec6bb3337ddb80fccba279bcd320edea49872dcc8"} Nov 01 15:10:06.655034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.655013 5473 generic.go:356] "Generic (PLEG): container finished" podID="0cd91561-4810-47d6-a24c-a6ee5a97d9f8" containerID="9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8" exitCode=0 Nov 01 15:10:06.655118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.655063 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" Nov 01 15:10:06.655118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.655089 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" event={"ID":"0cd91561-4810-47d6-a24c-a6ee5a97d9f8","Type":"ContainerDied","Data":"9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8"} Nov 01 15:10:06.655240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.655128 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7" event={"ID":"0cd91561-4810-47d6-a24c-a6ee5a97d9f8","Type":"ContainerDied","Data":"017d47332fd05b5f0adcdacdb0e4e3a82416c95033f886c0055613d82d188399"} Nov 01 15:10:06.655240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.655146 5473 scope.go:122] "RemoveContainer" containerID="9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8" Nov 01 15:10:06.673359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.673339 5473 scope.go:122] "RemoveContainer" containerID="9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8" Nov 01 15:10:06.673600 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:06.673583 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8\": container with ID starting with 9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8 not found: ID does not exist" containerID="9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8" Nov 01 15:10:06.673659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.673607 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8"} err="failed to get container status \"9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8\": rpc error: code = NotFound desc = could not find container \"9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8\": container with ID starting with 9f36b9a053e8423de87b6f7a1299c76d9fbb26de4ef7556b22a325fe984633d8 not found: ID does not exist" Nov 01 15:10:06.748274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.748061 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:06.748274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.748096 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgnk7\" (UniqueName: \"kubernetes.io/projected/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-kube-api-access-tgnk7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:06.748274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.748114 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cd91561-4810-47d6-a24c-a6ee5a97d9f8-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:06.768444 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:06.768381 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-e7ce08\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:06.882426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.882395 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7"] Nov 01 15:10:06.886120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:06.886099 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-52fffa/auth-test-model-52fffa-kserve-844fb56b85-zxnq7"] Nov 01 15:10:07.082589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:07.082516 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cd91561-4810-47d6-a24c-a6ee5a97d9f8" path="/var/lib/kubelet/pods/0cd91561-4810-47d6-a24c-a6ee5a97d9f8/volumes" Nov 01 15:10:07.664777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:07.664751 5473 generic.go:356] "Generic (PLEG): container finished" podID="f9843443-a8bf-4e3f-8cb5-da078eab35da" containerID="08e4939ed654accabceba68ec248f4d56eeabf7942bcadb1376f5be1d2e35547" exitCode=0 Nov 01 15:10:07.665105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:07.664831 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" event={"ID":"f9843443-a8bf-4e3f-8cb5-da078eab35da","Type":"ContainerDied","Data":"08e4939ed654accabceba68ec248f4d56eeabf7942bcadb1376f5be1d2e35547"} Nov 01 15:10:07.886388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:07.886364 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:07.924971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:07.924911 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:08.064044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.064007 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74\" (UniqueName: \"kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74\") pod \"36a4a488-2a74-4219-949a-2b13a2590f1a\" (UID: \"36a4a488-2a74-4219-949a-2b13a2590f1a\") " Nov 01 15:10:08.066103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.066075 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74" pod "36a4a488-2a74-4219-949a-2b13a2590f1a" (UID: "36a4a488-2a74-4219-949a-2b13a2590f1a"). InnerVolumeSpecName "kube-api-access-xbj74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:08.112916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.112889 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:08.165969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.165938 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbj74\" (UniqueName: \"kubernetes.io/projected/36a4a488-2a74-4219-949a-2b13a2590f1a-kube-api-access-xbj74\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:08.292107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.292073 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:08.673578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.673490 5473 generic.go:356] "Generic (PLEG): container finished" podID="36a4a488-2a74-4219-949a-2b13a2590f1a" containerID="4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457" exitCode=137 Nov 01 15:10:08.673578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.673543 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:08.673578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.673558 5473 scope.go:122] "RemoveContainer" containerID="4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457" Nov 01 15:10:08.685177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.685156 5473 scope.go:122] "RemoveContainer" containerID="4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457" Nov 01 15:10:08.685604 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:08.685568 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457\": container with ID starting with 4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457 not found: ID does not exist" containerID="4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457" Nov 01 15:10:08.685774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.685614 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457"} err="failed to get container status \"4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457\": rpc error: code = NotFound desc = could not find container \"4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457\": container with ID starting with 4a59dbefeef2ef45ecdbf8906adcb3b11fcb9afacc301229eac19b702ea60457 not found: ID does not exist" Nov 01 15:10:08.734018 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:08.733703 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-e7ce08\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:08.823203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.823179 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:08.874759 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:08.874721 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-e7ce08\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" pod="opendatahub/test-curl-7384-e7ce08" Nov 01 15:10:08.976269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.976179 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca\") pod \"f9843443-a8bf-4e3f-8cb5-da078eab35da\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " Nov 01 15:10:08.976413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.976328 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w\" (UniqueName: \"kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w\") pod \"f9843443-a8bf-4e3f-8cb5-da078eab35da\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " Nov 01 15:10:08.976413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.976406 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca\") pod \"f9843443-a8bf-4e3f-8cb5-da078eab35da\" (UID: \"f9843443-a8bf-4e3f-8cb5-da078eab35da\") " Nov 01 15:10:08.976547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.976519 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca" pod "f9843443-a8bf-4e3f-8cb5-da078eab35da" (UID: "f9843443-a8bf-4e3f-8cb5-da078eab35da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:10:08.976765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.976748 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:08.978433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.978411 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca" pod "f9843443-a8bf-4e3f-8cb5-da078eab35da" (UID: "f9843443-a8bf-4e3f-8cb5-da078eab35da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:10:08.978541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:08.978450 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w" pod "f9843443-a8bf-4e3f-8cb5-da078eab35da" (UID: "f9843443-a8bf-4e3f-8cb5-da078eab35da"). InnerVolumeSpecName "kube-api-access-jn77w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:09.077533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.077500 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn77w\" (UniqueName: \"kubernetes.io/projected/f9843443-a8bf-4e3f-8cb5-da078eab35da-kube-api-access-jn77w\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:09.077533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.077529 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9843443-a8bf-4e3f-8cb5-da078eab35da-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:09.082719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.082690 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a4a488-2a74-4219-949a-2b13a2590f1a" path="/var/lib/kubelet/pods/36a4a488-2a74-4219-949a-2b13a2590f1a/volumes" Nov 01 15:10:09.368381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.368344 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:09.570201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.570161 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:09.682926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.682847 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" Nov 01 15:10:09.682926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.682878 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n" event={"ID":"f9843443-a8bf-4e3f-8cb5-da078eab35da","Type":"ContainerDied","Data":"500cb72728626d8d9f5d98dec6bb3337ddb80fccba279bcd320edea49872dcc8"} Nov 01 15:10:09.683483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:09.682929 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500cb72728626d8d9f5d98dec6bb3337ddb80fccba279bcd320edea49872dcc8" Nov 01 15:10:10.195816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.195777 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg"] Nov 01 15:10:10.197308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.197277 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0cd91561-4810-47d6-a24c-a6ee5a97d9f8" containerName="main" Nov 01 15:10:10.197487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.197473 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cd91561-4810-47d6-a24c-a6ee5a97d9f8" containerName="main" Nov 01 15:10:10.197581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.197571 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f9843443-a8bf-4e3f-8cb5-da078eab35da" containerName="revoke-keys" Nov 01 15:10:10.197665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.197654 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9843443-a8bf-4e3f-8cb5-da078eab35da" containerName="revoke-keys" Nov 01 15:10:10.198122 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.198101 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cd91561-4810-47d6-a24c-a6ee5a97d9f8" containerName="main" Nov 01 15:10:10.198267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.198256 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9843443-a8bf-4e3f-8cb5-da078eab35da" containerName="revoke-keys" Nov 01 15:10:10.203412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.203389 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.256993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.256963 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg"] Nov 01 15:10:10.283009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.282980 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2cf1ee-openshift-default-dockercfg-9jpjk\"" Nov 01 15:10:10.394508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394467 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn9z\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.394707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394592 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.394707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394656 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.394811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394730 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.394811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394787 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.394925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394821 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.394973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.394912 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.395025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.395009 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.395070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.395049 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.445066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.445030 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:10.495799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495715 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn9z\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.495799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495762 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.495799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495795 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495815 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495854 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495892 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.495918 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496005 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496032 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496341 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496400 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496463 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496493 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.496701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.496659 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.498488 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.498461 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.498901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.498879 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.554123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.554087 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.554529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.554511 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn9z\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z\") pod \"e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.633249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.633197 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:10.817799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.817763 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:10.874626 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:10.871849 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:11.236239 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:11.236192 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce384e4_cde3_4258_9484_45bcd3a10cea.slice/crio-6ec5e99351a61b75b52359242c29990983d7826d9d9fdb0a37a3cd780d04e446 WatchSource:0}: Error finding container 6ec5e99351a61b75b52359242c29990983d7826d9d9fdb0a37a3cd780d04e446: Status 404 returned error can't find the container with id 6ec5e99351a61b75b52359242c29990983d7826d9d9fdb0a37a3cd780d04e446 Nov 01 15:10:11.239279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.239243 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:10:11.239364 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.239319 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:10:11.239364 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.239349 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:10:11.239480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.239459 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg"] Nov 01 15:10:11.364058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.364027 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:11.698123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.698087 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" event={"ID":"cce384e4-cde3-4258-9484-45bcd3a10cea","Type":"ContainerStarted","Data":"73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498"} Nov 01 15:10:11.698330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.698145 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" event={"ID":"cce384e4-cde3-4258-9484-45bcd3a10cea","Type":"ContainerStarted","Data":"6ec5e99351a61b75b52359242c29990983d7826d9d9fdb0a37a3cd780d04e446"} Nov 01 15:10:11.815539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.815484 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podStartSLOduration=1.815472752 podStartE2EDuration="1.815472752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:11.815184852 +0000 UTC m=+3150.873336702" watchObservedRunningTime="2026-11-01 15:10:11.815472752 +0000 UTC m=+3150.873624601" Nov 01 15:10:11.818204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:11.818176 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:12.824765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:12.824734 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:13.712638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:13.712603 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:13.713644 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:13.713618 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:10:15.262112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.262057 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:15.507435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.507366 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:15.644654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.644574 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9"] Nov 01 15:10:15.648931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.648907 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.679952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.679924 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:15.741236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.730150 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9"] Nov 01 15:10:15.762201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.755505 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0f1e68-dockercfg-zzqzq\"" Nov 01 15:10:15.762201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.755588 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0f1e68\"" Nov 01 15:10:15.764662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.763861 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vbt\" (UniqueName: \"kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt\") pod \"payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.764662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.764010 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.809942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.809908 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n"] Nov 01 15:10:15.814049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.814028 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:15.865294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.865266 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vbt\" (UniqueName: \"kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt\") pod \"payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.865437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.865339 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghb2\" (UniqueName: \"kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2\") pod \"payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:15.865437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.865374 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:15.865437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.865427 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.866115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.866090 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.871702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.871678 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n"] Nov 01 15:10:15.872017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.871998 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:15.923547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.923482 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vbt\" (UniqueName: \"kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt\") pod \"payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.963601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.963572 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:15.966083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.966061 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghb2\" (UniqueName: \"kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2\") pod \"payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:15.966157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.966094 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:15.966919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:15.966764 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:16.043146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.043065 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghb2\" (UniqueName: \"kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2\") pod \"payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:16.046475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.046403 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78"] Nov 01 15:10:16.052653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.052624 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.104169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.104136 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78"] Nov 01 15:10:16.107210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.107124 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:16.108187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.108165 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:10:16.108187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.108168 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:10:16.129086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.128578 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:16.168540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.168505 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgmr8\" (UniqueName: \"kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.168704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.168588 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.168888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.168857 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.269852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.269818 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgmr8\" (UniqueName: \"kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.270234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.269876 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.270234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.269924 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.270545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.270525 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.272308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.272289 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.327287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.327265 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgmr8\" (UniqueName: \"kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8\") pod \"maas-api-revoke-sub-71ae9a22-xlj78\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.337480 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:16.337453 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fb9ec5_b4cb_4969_93fc_5e7d0d4d9e8b.slice/crio-2e68086d5ae46b7f05114c8bff3dd128d5448c0ffb08f8e10e10b506637882ac WatchSource:0}: Error finding container 2e68086d5ae46b7f05114c8bff3dd128d5448c0ffb08f8e10e10b506637882ac: Status 404 returned error can't find the container with id 2e68086d5ae46b7f05114c8bff3dd128d5448c0ffb08f8e10e10b506637882ac Nov 01 15:10:16.340715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.340688 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9"] Nov 01 15:10:16.372992 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.372974 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:16.520029 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:16.520004 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab1f1be_9eb6_4056_9012_711080246644.slice/crio-413ca216a549e691188152eb5a1174611d5317c232589d0c4dd1571929b44085 WatchSource:0}: Error finding container 413ca216a549e691188152eb5a1174611d5317c232589d0c4dd1571929b44085: Status 404 returned error can't find the container with id 413ca216a549e691188152eb5a1174611d5317c232589d0c4dd1571929b44085 Nov 01 15:10:16.523786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.523765 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n"] Nov 01 15:10:16.746821 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:16.746790 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef18acc_c96e_4579_9652_794c0daec4b9.slice/crio-4fc73e7952751cd64d495d2cd1b7474ff2571a10572dee0797d571f0ccd910ff WatchSource:0}: Error finding container 4fc73e7952751cd64d495d2cd1b7474ff2571a10572dee0797d571f0ccd910ff: Status 404 returned error can't find the container with id 4fc73e7952751cd64d495d2cd1b7474ff2571a10572dee0797d571f0ccd910ff Nov 01 15:10:16.747663 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.747638 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" event={"ID":"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b","Type":"ContainerStarted","Data":"b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed"} Nov 01 15:10:16.747744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.747686 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" event={"ID":"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b","Type":"ContainerStarted","Data":"2e68086d5ae46b7f05114c8bff3dd128d5448c0ffb08f8e10e10b506637882ac"} Nov 01 15:10:16.747874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.747774 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:16.748744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.748720 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78"] Nov 01 15:10:16.749505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.749481 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" event={"ID":"6ab1f1be-9eb6-4056-9012-711080246644","Type":"ContainerStarted","Data":"27f0037d9bca4f80431b11a355ffaa4e14656ee6f6feb798d2b3978300b17ff1"} Nov 01 15:10:16.749578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.749530 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:16.749578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.749558 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" event={"ID":"6ab1f1be-9eb6-4056-9012-711080246644","Type":"ContainerStarted","Data":"413ca216a549e691188152eb5a1174611d5317c232589d0c4dd1571929b44085"} Nov 01 15:10:16.865812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.865760 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" podStartSLOduration=1.8657423 podStartE2EDuration="1.8657423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:16.860941224 +0000 UTC m=+3155.919093085" watchObservedRunningTime="2026-11-01 15:10:16.8657423 +0000 UTC m=+3155.923894151" Nov 01 15:10:16.977386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:16.977300 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" podStartSLOduration=1.977280193 podStartE2EDuration="1.977280193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:16.974380452 +0000 UTC m=+3156.032532303" watchObservedRunningTime="2026-11-01 15:10:16.977280193 +0000 UTC m=+3156.035432044" Nov 01 15:10:17.757699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:17.757665 5473 generic.go:356] "Generic (PLEG): container finished" podID="1ef18acc-c96e-4579-9652-794c0daec4b9" containerID="8dabdc63666c192e044102e28b2fe750da8634e168e62874c081f1fb83707459" exitCode=0 Nov 01 15:10:17.758134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:17.757755 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" event={"ID":"1ef18acc-c96e-4579-9652-794c0daec4b9","Type":"ContainerDied","Data":"8dabdc63666c192e044102e28b2fe750da8634e168e62874c081f1fb83707459"} Nov 01 15:10:17.758134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:17.757820 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" event={"ID":"1ef18acc-c96e-4579-9652-794c0daec4b9","Type":"ContainerStarted","Data":"4fc73e7952751cd64d495d2cd1b7474ff2571a10572dee0797d571f0ccd910ff"} Nov 01 15:10:18.034778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:18.034751 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:18.913023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:18.912995 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:19.001861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.001836 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca\") pod \"1ef18acc-c96e-4579-9652-794c0daec4b9\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " Nov 01 15:10:19.002048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.002033 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca\") pod \"1ef18acc-c96e-4579-9652-794c0daec4b9\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " Nov 01 15:10:19.002129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.002063 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8\" (UniqueName: \"kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8\") pod \"1ef18acc-c96e-4579-9652-794c0daec4b9\" (UID: \"1ef18acc-c96e-4579-9652-794c0daec4b9\") " Nov 01 15:10:19.002378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.002357 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca" pod "1ef18acc-c96e-4579-9652-794c0daec4b9" (UID: "1ef18acc-c96e-4579-9652-794c0daec4b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:10:19.003860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.003841 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca" pod "1ef18acc-c96e-4579-9652-794c0daec4b9" (UID: "1ef18acc-c96e-4579-9652-794c0daec4b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:10:19.004076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.004056 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8" pod "1ef18acc-c96e-4579-9652-794c0daec4b9" (UID: "1ef18acc-c96e-4579-9652-794c0daec4b9"). InnerVolumeSpecName "kube-api-access-rgmr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:10:19.103671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.103640 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:19.103671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.103668 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgmr8\" (UniqueName: \"kubernetes.io/projected/1ef18acc-c96e-4579-9652-794c0daec4b9-kube-api-access-rgmr8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:19.103848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.103680 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef18acc-c96e-4579-9652-794c0daec4b9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:10:19.426728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.426649 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:19.777168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.777134 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" Nov 01 15:10:19.777168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.777139 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78" event={"ID":"1ef18acc-c96e-4579-9652-794c0daec4b9","Type":"ContainerDied","Data":"4fc73e7952751cd64d495d2cd1b7474ff2571a10572dee0797d571f0ccd910ff"} Nov 01 15:10:19.777420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.777193 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fc73e7952751cd64d495d2cd1b7474ff2571a10572dee0797d571f0ccd910ff" Nov 01 15:10:19.865589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:19.865552 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:20.061078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.060995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:20.232371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.232325 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:20.356117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.355996 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv"] Nov 01 15:10:20.356902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.356872 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1ef18acc-c96e-4579-9652-794c0daec4b9" containerName="revoke-keys" Nov 01 15:10:20.357035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.356903 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef18acc-c96e-4579-9652-794c0daec4b9" containerName="revoke-keys" Nov 01 15:10:20.357155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.357137 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef18acc-c96e-4579-9652-794c0daec4b9" containerName="revoke-keys" Nov 01 15:10:20.362917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.362895 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.405361 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.405330 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv"] Nov 01 15:10:20.420202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.420173 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.420434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.420414 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ff84\" (UniqueName: \"kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84\") pod \"payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.426836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.426812 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2cf1ee-dockercfg-64np5\"" Nov 01 15:10:20.427006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.426992 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2cf1ee\"" Nov 01 15:10:20.498966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.498926 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:20.521575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.521539 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ff84\" (UniqueName: \"kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84\") pod \"payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.521799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.521642 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.522404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.522383 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.579658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.579622 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ff84\" (UniqueName: \"kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84\") pod \"payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:20.680483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:20.680396 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:21.052451 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:10:21.052418 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5540bee_3f36_487a_bd82_dbcf44e15c2b.slice/crio-f7332d6ed70fd8b168f24682ec847f09afd2e5b3fd50431d6ccf8952dab83a8e WatchSource:0}: Error finding container f7332d6ed70fd8b168f24682ec847f09afd2e5b3fd50431d6ccf8952dab83a8e: Status 404 returned error can't find the container with id f7332d6ed70fd8b168f24682ec847f09afd2e5b3fd50431d6ccf8952dab83a8e Nov 01 15:10:21.053098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:21.053072 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv"] Nov 01 15:10:21.152929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:21.152898 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:21.795418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:21.795381 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" event={"ID":"f5540bee-3f36-487a-bd82-dbcf44e15c2b","Type":"ContainerStarted","Data":"b38b16b206a3d9a1766ddc368cf838c6167a8aa783ba757830cb98a08ed210fa"} Nov 01 15:10:21.795622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:21.795435 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" event={"ID":"f5540bee-3f36-487a-bd82-dbcf44e15c2b","Type":"ContainerStarted","Data":"f7332d6ed70fd8b168f24682ec847f09afd2e5b3fd50431d6ccf8952dab83a8e"} Nov 01 15:10:21.795622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:21.795466 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:21.910399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:21.910346 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" podStartSLOduration=1.910328336 podStartE2EDuration="1.910328336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:10:21.909144167 +0000 UTC m=+3160.967296017" watchObservedRunningTime="2026-11-01 15:10:21.910328336 +0000 UTC m=+3160.968480185" Nov 01 15:10:24.630339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:24.630308 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:24.840967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:24.840935 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:25.981931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:25.981673 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:27.760688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:27.760660 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:10:27.761135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:27.760768 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:10:28.691919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:28.691890 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:28.890813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:28.890778 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:30.739733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:30.739698 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:31.108418 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:10:31.108374 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:10:32.805413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:32.805384 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:10:35.212360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:35.212326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:39.694893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:39.694852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:44.092916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:44.092871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:48.531433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:48.531390 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:49.070508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:49.070476 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:52.935112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:52.935076 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:57.301465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:57.301429 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:10:59.947559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:10:59.947524 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:01.109005 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:11:01.108968 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:11:01.809533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:01.809501 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:06.210036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:06.209997 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:10.562793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:10.562763 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:10.731200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:10.731135 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:10.898123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:10.898039 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:15.143669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:15.143637 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:19.616675 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:19.616638 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:21.017410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.017367 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:21.475129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.475050 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj"] Nov 01 15:11:21.479863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.479842 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.554718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.554688 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-926972-openshift-default-dockercfg-kpzf9\"" Nov 01 15:11:21.576692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.576655 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.576692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.576702 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhb4m\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.576940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.576768 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.577006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.576933 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.577006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.576995 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.577176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.577109 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.577255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.577203 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.577322 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.577269 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.577322 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.577296 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.586279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.586253 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj"] Nov 01 15:11:21.605652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.605609 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:21.678929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.678630 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.678929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.678723 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.678929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.678773 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.678929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.678800 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.679400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.678894 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.679559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.679539 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhb4m\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.679683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.679669 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.679808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.679782 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.681436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.679859 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.681670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.681650 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.681884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.681153 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.682013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.681384 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.682105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.680624 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.682481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.682460 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.683632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.683608 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.690085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.690042 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.742650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.742572 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.742789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.742716 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhb4m\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m\") pod \"e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:21.794306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:21.794277 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:22.070208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:22.070161 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:22.210879 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:11:22.210850 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abf39f2_78ae_44ce_801b_dcaab11f5802.slice/crio-475c5129b713d87e74ba9b6843efcc0c53393f12d59ebbd8e34682708bf35427 WatchSource:0}: Error finding container 475c5129b713d87e74ba9b6843efcc0c53393f12d59ebbd8e34682708bf35427: Status 404 returned error can't find the container with id 475c5129b713d87e74ba9b6843efcc0c53393f12d59ebbd8e34682708bf35427 Nov 01 15:11:22.213938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:22.213901 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:11:22.214040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:22.213989 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:11:22.214040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:22.214031 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:11:22.215388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:22.215365 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj"] Nov 01 15:11:22.250077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:22.250046 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" event={"ID":"6abf39f2-78ae-44ce-801b-dcaab11f5802","Type":"ContainerStarted","Data":"475c5129b713d87e74ba9b6843efcc0c53393f12d59ebbd8e34682708bf35427"} Nov 01 15:11:23.259425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:23.259386 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" event={"ID":"6abf39f2-78ae-44ce-801b-dcaab11f5802","Type":"ContainerStarted","Data":"a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514"} Nov 01 15:11:23.378746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:23.378696 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podStartSLOduration=2.378678301 podStartE2EDuration="2.378678301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:23.377585548 +0000 UTC m=+3222.435737397" watchObservedRunningTime="2026-11-01 15:11:23.378678301 +0000 UTC m=+3222.436830151" Nov 01 15:11:23.794670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:23.794640 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:23.800451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:23.800425 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:24.045962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:24.045874 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:24.212582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:24.212549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:24.267450 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:24.267416 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:24.268605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:24.268584 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:11:24.371162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:24.371082 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:26.081113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:26.081076 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:26.269163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:26.269131 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:26.450434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:26.450335 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:28.200085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:28.200048 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7"] Nov 01 15:11:28.249039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:28.248983 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:29.013922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:29.013890 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:31.109227 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:11:31.109177 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:11:33.615658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:33.615626 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:35.358206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.358169 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:35.533153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.533116 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:35.751910 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.751803 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5"] Nov 01 15:11:35.757717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.757691 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:35.803352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.803322 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5"] Nov 01 15:11:35.858240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.848512 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-926972\"" Nov 01 15:11:35.858240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.848668 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-926972-dockercfg-rgtxh\"" Nov 01 15:11:35.904945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.904905 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:35.905143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.905004 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkttv\" (UniqueName: \"kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv\") pod \"payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:35.917270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.915719 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8"] Nov 01 15:11:35.920794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.920772 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:35.970681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:35.970652 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8"] Nov 01 15:11:36.006734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.006644 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:36.006734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.006728 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz64c\" (UniqueName: \"kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c\") pod \"payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.006942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.006763 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkttv\" (UniqueName: \"kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv\") pod \"payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:36.006942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.006892 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.007372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.007352 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:36.064744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.064699 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkttv\" (UniqueName: \"kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv\") pod \"payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:36.073578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.073549 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:36.108985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.108942 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.109287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.109266 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz64c\" (UniqueName: \"kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c\") pod \"payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.110793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.110766 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.173455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.173383 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz64c\" (UniqueName: \"kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c\") pod \"payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.264609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.264583 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:36.469672 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:11:36.469642 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253446fe_1c2b_4edd_a0d1_08b2a0a36b7c.slice/crio-1c08f83744b1306d73a0b9971a53ec901cae094ab84b3bad2797efaac7122de7 WatchSource:0}: Error finding container 1c08f83744b1306d73a0b9971a53ec901cae094ab84b3bad2797efaac7122de7: Status 404 returned error can't find the container with id 1c08f83744b1306d73a0b9971a53ec901cae094ab84b3bad2797efaac7122de7 Nov 01 15:11:36.473192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.473107 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5"] Nov 01 15:11:36.654718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:36.654691 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8"] Nov 01 15:11:36.657272 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:11:36.657242 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4bbb61_4734_4502_b0d0_27f3b2ef4958.slice/crio-a4cbc7b6ee623409feb0c397846d9e2328138bda0a35811fde101dee16a3d640 WatchSource:0}: Error finding container a4cbc7b6ee623409feb0c397846d9e2328138bda0a35811fde101dee16a3d640: Status 404 returned error can't find the container with id a4cbc7b6ee623409feb0c397846d9e2328138bda0a35811fde101dee16a3d640 Nov 01 15:11:37.412802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.412759 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" event={"ID":"8e4bbb61-4734-4502-b0d0-27f3b2ef4958","Type":"ContainerStarted","Data":"50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730"} Nov 01 15:11:37.413006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.412835 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" event={"ID":"8e4bbb61-4734-4502-b0d0-27f3b2ef4958","Type":"ContainerStarted","Data":"a4cbc7b6ee623409feb0c397846d9e2328138bda0a35811fde101dee16a3d640"} Nov 01 15:11:37.413075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.413015 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:37.414590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.414557 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" event={"ID":"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c","Type":"ContainerStarted","Data":"b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d"} Nov 01 15:11:37.414736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.414601 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" event={"ID":"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c","Type":"ContainerStarted","Data":"1c08f83744b1306d73a0b9971a53ec901cae094ab84b3bad2797efaac7122de7"} Nov 01 15:11:37.414736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.414669 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:37.528967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.528917 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" podStartSLOduration=2.528898151 podStartE2EDuration="2.528898151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:37.526005077 +0000 UTC m=+3236.584156929" watchObservedRunningTime="2026-11-01 15:11:37.528898151 +0000 UTC m=+3236.587050000" Nov 01 15:11:37.652305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:37.652244 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" podStartSLOduration=2.652226303 podStartE2EDuration="2.652226303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:37.648647443 +0000 UTC m=+3236.706799316" watchObservedRunningTime="2026-11-01 15:11:37.652226303 +0000 UTC m=+3236.710378145" Nov 01 15:11:38.030676 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:38.030642 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:38.201665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:38.201621 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:38.377271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:38.377164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:40.438420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:40.438330 5473 generic.go:356] "Generic (PLEG): container finished" podID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerID="9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339" exitCode=0 Nov 01 15:11:40.438420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:40.438370 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" event={"ID":"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a","Type":"ContainerDied","Data":"9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339"} Nov 01 15:11:40.438936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:40.438705 5473 scope.go:122] "RemoveContainer" containerID="9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339" Nov 01 15:11:40.440197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:40.440169 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:11:40.440306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:40.440254 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:11:40.440306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:40.440289 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:11:41.458159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:41.458124 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" event={"ID":"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a","Type":"ContainerStarted","Data":"0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02"} Nov 01 15:11:41.791210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:41.791173 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:11:41.791428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:41.791255 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 15:11:41.791428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:41.791320 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 15:11:42.625244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:42.624525 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:42.792280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:42.791340 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 15:11:42.792280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:42.791395 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 15:11:43.220866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.220829 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" podUID="4981ac6e-16d1-4e53-b63b-8066959cecf7" containerName="main" containerID="cri-o://ee4c9f22c14c556d58eb2888abd9e955c84e034473f697d97e9b4b252103c082" gracePeriod=45 Nov 01 15:11:43.476750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.476464 5473 generic.go:356] "Generic (PLEG): container finished" podID="4981ac6e-16d1-4e53-b63b-8066959cecf7" containerID="ee4c9f22c14c556d58eb2888abd9e955c84e034473f697d97e9b4b252103c082" exitCode=0 Nov 01 15:11:43.476750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.476671 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" event={"ID":"4981ac6e-16d1-4e53-b63b-8066959cecf7","Type":"ContainerDied","Data":"ee4c9f22c14c556d58eb2888abd9e955c84e034473f697d97e9b4b252103c082"} Nov 01 15:11:43.548238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.533377 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:11:43.616892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.616862 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd\" (UniqueName: \"kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd\") pod \"4981ac6e-16d1-4e53-b63b-8066959cecf7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " Nov 01 15:11:43.617087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.616938 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache\") pod \"4981ac6e-16d1-4e53-b63b-8066959cecf7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " Nov 01 15:11:43.617087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617027 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs\") pod \"4981ac6e-16d1-4e53-b63b-8066959cecf7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " Nov 01 15:11:43.617087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617072 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm\") pod \"4981ac6e-16d1-4e53-b63b-8066959cecf7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " Nov 01 15:11:43.617356 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617093 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir\") pod \"4981ac6e-16d1-4e53-b63b-8066959cecf7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " Nov 01 15:11:43.617356 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617110 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home\") pod \"4981ac6e-16d1-4e53-b63b-8066959cecf7\" (UID: \"4981ac6e-16d1-4e53-b63b-8066959cecf7\") " Nov 01 15:11:43.617356 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617182 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache" pod "4981ac6e-16d1-4e53-b63b-8066959cecf7" (UID: "4981ac6e-16d1-4e53-b63b-8066959cecf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:11:43.617533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617365 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir" pod "4981ac6e-16d1-4e53-b63b-8066959cecf7" (UID: "4981ac6e-16d1-4e53-b63b-8066959cecf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:11:43.617533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617396 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home" pod "4981ac6e-16d1-4e53-b63b-8066959cecf7" (UID: "4981ac6e-16d1-4e53-b63b-8066959cecf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:11:43.617719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617699 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:11:43.617719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617721 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:11:43.617864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.617732 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:11:43.619309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.619290 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd" pod "4981ac6e-16d1-4e53-b63b-8066959cecf7" (UID: "4981ac6e-16d1-4e53-b63b-8066959cecf7"). InnerVolumeSpecName "kube-api-access-lzsjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:11:43.619496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.619467 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm" pod "4981ac6e-16d1-4e53-b63b-8066959cecf7" (UID: "4981ac6e-16d1-4e53-b63b-8066959cecf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:11:43.619608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.619531 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs" pod "4981ac6e-16d1-4e53-b63b-8066959cecf7" (UID: "4981ac6e-16d1-4e53-b63b-8066959cecf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:11:43.719125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.719096 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzsjd\" (UniqueName: \"kubernetes.io/projected/4981ac6e-16d1-4e53-b63b-8066959cecf7-kube-api-access-lzsjd\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:11:43.719125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.719124 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4981ac6e-16d1-4e53-b63b-8066959cecf7-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:11:43.719541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.719133 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4981ac6e-16d1-4e53-b63b-8066959cecf7-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:11:43.790757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.790726 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 15:11:43.790916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:43.790781 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 15:11:44.486560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.486529 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" Nov 01 15:11:44.486560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.486544 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7" event={"ID":"4981ac6e-16d1-4e53-b63b-8066959cecf7","Type":"ContainerDied","Data":"c1494b30e4a134139dd50a20bcb6284d33fbde278883d38246b8bca1276c3bf8"} Nov 01 15:11:44.486833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.486601 5473 scope.go:122] "RemoveContainer" containerID="ee4c9f22c14c556d58eb2888abd9e955c84e034473f697d97e9b4b252103c082" Nov 01 15:11:44.658746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.658710 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7"] Nov 01 15:11:44.708390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.708360 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b3f06c/auth-test-model-b3f06c-kserve-85768d54d5-472w7"] Nov 01 15:11:44.791135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.791104 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 15:11:44.791529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:44.791154 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 15:11:45.083358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:45.083269 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4981ac6e-16d1-4e53-b63b-8066959cecf7" path="/var/lib/kubelet/pods/4981ac6e-16d1-4e53-b63b-8066959cecf7/volumes" Nov 01 15:11:45.791535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:45.791502 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 01 15:11:45.791930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:45.791552 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 01 15:11:46.795468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:46.795436 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:11:46.795991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:46.795811 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:11:46.796425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:46.796407 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:11:47.055367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:47.055283 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:48.426240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:48.426192 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:11:48.426626 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:48.426290 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:11:51.517154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:51.517075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:54.830740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.830704 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh"] Nov 01 15:11:54.831379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.831363 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4981ac6e-16d1-4e53-b63b-8066959cecf7" containerName="main" Nov 01 15:11:54.831434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.831382 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4981ac6e-16d1-4e53-b63b-8066959cecf7" containerName="main" Nov 01 15:11:54.831542 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.831532 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4981ac6e-16d1-4e53-b63b-8066959cecf7" containerName="main" Nov 01 15:11:54.836577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.836557 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.881547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.881523 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.881699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.881562 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.881780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.881758 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2b5\" (UniqueName: \"kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.882105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.881813 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.882105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.881896 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.882105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.881931 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.888974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.888951 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh"] Nov 01 15:11:54.934983 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.934957 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"kube-root-ca.crt\"" Nov 01 15:11:54.935089 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.934957 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"openshift-service-ca.crt\"" Nov 01 15:11:54.935359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.935345 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"default-dockercfg-tbqbg\"" Nov 01 15:11:54.941124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.941103 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-8fd363\"/\"auth-test-model-8fd363-kserve-self-signed-certs\"" Nov 01 15:11:54.982831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.982807 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.982953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.982841 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.982953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.982911 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2b5\" (UniqueName: \"kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.982953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.982946 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.983085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.982976 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.983085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.983007 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.983377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.983353 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.983481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.983380 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.983481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.983431 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.985040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.985019 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:54.985702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:54.985681 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:55.040275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:55.040248 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:55.040518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:55.040500 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2b5\" (UniqueName: \"kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5\") pod \"auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:55.149161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:55.149078 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:11:55.521633 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:11:55.521606 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d3fe657_30d8_45b6_82f0_91b3532227ea.slice/crio-465a4085498fa76761eac0f50e283afbec530b5441dba2408634edeba9f583cf WatchSource:0}: Error finding container 465a4085498fa76761eac0f50e283afbec530b5441dba2408634edeba9f583cf: Status 404 returned error can't find the container with id 465a4085498fa76761eac0f50e283afbec530b5441dba2408634edeba9f583cf Nov 01 15:11:55.524602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:55.524580 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh"] Nov 01 15:11:55.579242 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:55.579204 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" event={"ID":"0d3fe657-30d8-45b6-82f0-91b3532227ea","Type":"ContainerStarted","Data":"465a4085498fa76761eac0f50e283afbec530b5441dba2408634edeba9f583cf"} Nov 01 15:11:56.064415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:56.064383 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:11:56.588853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:56.588818 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" event={"ID":"0d3fe657-30d8-45b6-82f0-91b3532227ea","Type":"ContainerStarted","Data":"cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d"} Nov 01 15:11:56.708654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:56.708603 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" podStartSLOduration=2.7085889229999998 podStartE2EDuration="2.708588923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:11:56.705250386 +0000 UTC m=+3255.763402237" watchObservedRunningTime="2026-11-01 15:11:56.708588923 +0000 UTC m=+3255.766740773" Nov 01 15:11:57.149352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:11:57.149317 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:12:00.274965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:00.274931 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:01.108460 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:12:01.108422 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:12:01.867737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:01.867701 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:04.887935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:04.887901 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:05.149394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:05.149303 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:12:05.161891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:05.161864 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:12:05.673793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:05.673765 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:12:09.377590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:09.377559 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:13.884165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:13.884130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:18.337892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:18.337860 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:22.814586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:22.814557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:27.236012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:27.235972 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:31.115403 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:12:31.115360 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:12:31.718734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:31.718701 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:34.843643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:34.843612 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:35.026326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:35.026294 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:36.082399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:36.082371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:36.476330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:36.476236 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:37.351200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.351171 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:37.801309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.801278 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v"] Nov 01 15:12:37.806042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.806026 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.864988 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-15ad1e-openshift-default-dockercfg-sddmv\"" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.878979 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5fn\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879069 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879115 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879152 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879258 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879396 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.888259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.879513 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.921280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.918783 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:37.921280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.920636 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v"] Nov 01 15:12:37.980536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980494 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.980745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980550 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.980745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980576 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.980745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980631 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.980745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980658 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.980745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980734 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980801 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980838 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980875 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5fn\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.980924 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.981022 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.981135 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.981354 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.981406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.981394 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.983069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.983048 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:37.983206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:37.983190 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:38.039090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.039056 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5fn\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:38.039758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.039740 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token\") pod \"e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:38.120713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.120649 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:38.190379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.190348 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:38.562770 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:12:38.562734 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75df714_1cda_42e8_a61f_da2de949f015.slice/crio-b9c4e4f760a15f8347819d7edc3a1686b088d1e2ffe142d2bf83d8de647d3e38 WatchSource:0}: Error finding container b9c4e4f760a15f8347819d7edc3a1686b088d1e2ffe142d2bf83d8de647d3e38: Status 404 returned error can't find the container with id b9c4e4f760a15f8347819d7edc3a1686b088d1e2ffe142d2bf83d8de647d3e38 Nov 01 15:12:38.565611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.565576 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:12:38.565721 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.565647 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:12:38.565721 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.565675 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:12:38.566720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.566696 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v"] Nov 01 15:12:38.924859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.924759 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" event={"ID":"d75df714-1cda-42e8-a61f-da2de949f015","Type":"ContainerStarted","Data":"caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4"} Nov 01 15:12:38.924859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:38.924822 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" event={"ID":"d75df714-1cda-42e8-a61f-da2de949f015","Type":"ContainerStarted","Data":"b9c4e4f760a15f8347819d7edc3a1686b088d1e2ffe142d2bf83d8de647d3e38"} Nov 01 15:12:39.051107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:39.051048 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podStartSLOduration=2.051034912 podStartE2EDuration="2.051034912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:12:39.046450991 +0000 UTC m=+3298.104602841" watchObservedRunningTime="2026-11-01 15:12:39.051034912 +0000 UTC m=+3298.109186762" Nov 01 15:12:39.121189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:39.121150 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:39.128024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:39.127996 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:39.932355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:39.932323 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:39.933460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:39.933437 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:12:41.057414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:41.057375 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:41.816355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:41.816326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:42.002091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:42.002051 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:42.180692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:42.180613 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:44.101858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:44.101827 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:12:44.106581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:44.106562 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:12:44.930241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:44.930180 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:45.111498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:45.111462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:45.974385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:45.974339 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:46.150287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:46.150247 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:46.350879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:46.350844 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:47.549649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:47.549616 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:48.758239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:48.749764 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:49.371838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:49.371801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:49.553035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:49.553001 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:51.097551 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:51.097525 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:53.429256 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:53.429191 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:53.602870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:53.602833 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:55.045338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.045306 5473 generic.go:356] "Generic (PLEG): container finished" podID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerID="73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498" exitCode=0 Nov 01 15:12:55.045767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.045379 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" event={"ID":"cce384e4-cde3-4258-9484-45bcd3a10cea","Type":"ContainerDied","Data":"73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498"} Nov 01 15:12:55.045767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.045746 5473 scope.go:122] "RemoveContainer" containerID="73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498" Nov 01 15:12:55.047292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.047267 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:12:55.047362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.047335 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:12:55.047403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.047385 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:12:55.844666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:55.844633 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:56.061206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:56.061174 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" event={"ID":"cce384e4-cde3-4258-9484-45bcd3a10cea","Type":"ContainerStarted","Data":"043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896"} Nov 01 15:12:56.817974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:56.817940 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:12:56.818935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:56.818909 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 15:12:56.819075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:56.818964 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 15:12:57.818444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:57.818413 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 15:12:57.818818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:57.818468 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 15:12:58.148056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.147922 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:58.350448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.350410 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:12:58.521129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.521096 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx"] Nov 01 15:12:58.524197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.524181 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.654281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.595822 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjj5\" (UniqueName: \"kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5\") pod \"payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.654281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.596028 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.654281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.618541 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx"] Nov 01 15:12:58.654281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.636124 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e92165-dockercfg-84hhg\"" Nov 01 15:12:58.654281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.636187 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e92165\"" Nov 01 15:12:58.683332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.683304 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7"] Nov 01 15:12:58.686487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.686458 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.697640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.697603 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjj5\" (UniqueName: \"kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5\") pod \"payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.697778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.697686 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqnn\" (UniqueName: \"kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn\") pod \"payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.697778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.697767 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.697872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.697805 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.698603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.698577 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.712613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.712594 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:12:58.754977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.754941 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7"] Nov 01 15:12:58.756770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.756746 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjj5\" (UniqueName: \"kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5\") pod \"payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:58.799111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.799041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.799313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.799297 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqnn\" (UniqueName: \"kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn\") pod \"payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.799833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.799814 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.818675 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.818652 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 15:12:58.818977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.818691 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 15:12:58.856786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.856764 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqnn\" (UniqueName: \"kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn\") pod \"payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:58.895993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:58.895951 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:12:59.002052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:59.002020 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:12:59.284905 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:12:59.284875 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb49e057_732e_4399_b9c6_8e238f50e9b9.slice/crio-e3a49af79d12e46ca5a90fee32214152ee81c796123b05c1f25205f740349149 WatchSource:0}: Error finding container e3a49af79d12e46ca5a90fee32214152ee81c796123b05c1f25205f740349149: Status 404 returned error can't find the container with id e3a49af79d12e46ca5a90fee32214152ee81c796123b05c1f25205f740349149 Nov 01 15:12:59.289323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:59.289297 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx"] Nov 01 15:12:59.402230 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:12:59.402193 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8d1693_f28a_42c7_81c5_3577e2438f3a.slice/crio-57dc8c996ba51e139eb970e0e512f599e0dfffad58412e138ac4b9e4a545c5f9 WatchSource:0}: Error finding container 57dc8c996ba51e139eb970e0e512f599e0dfffad58412e138ac4b9e4a545c5f9: Status 404 returned error can't find the container with id 57dc8c996ba51e139eb970e0e512f599e0dfffad58412e138ac4b9e4a545c5f9 Nov 01 15:12:59.406209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:59.406187 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7"] Nov 01 15:12:59.818950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:59.818914 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 15:12:59.819452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:12:59.818967 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 15:13:00.095970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.095871 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" event={"ID":"4b8d1693-f28a-42c7-81c5-3577e2438f3a","Type":"ContainerStarted","Data":"6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526"} Nov 01 15:13:00.095970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.095925 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:13:00.095970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.095945 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" event={"ID":"4b8d1693-f28a-42c7-81c5-3577e2438f3a","Type":"ContainerStarted","Data":"57dc8c996ba51e139eb970e0e512f599e0dfffad58412e138ac4b9e4a545c5f9"} Nov 01 15:13:00.097672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.097647 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" event={"ID":"cb49e057-732e-4399-b9c6-8e238f50e9b9","Type":"ContainerStarted","Data":"0e76d79f1caa9f9a0a43fd01952f9a915748606aff7d639a7b7ed4c1edd90f30"} Nov 01 15:13:00.097835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.097688 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" event={"ID":"cb49e057-732e-4399-b9c6-8e238f50e9b9","Type":"ContainerStarted","Data":"e3a49af79d12e46ca5a90fee32214152ee81c796123b05c1f25205f740349149"} Nov 01 15:13:00.097835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.097717 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:13:00.213923 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.213873 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" podStartSLOduration=2.213859508 podStartE2EDuration="2.213859508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:13:00.211229028 +0000 UTC m=+3319.269380867" watchObservedRunningTime="2026-11-01 15:13:00.213859508 +0000 UTC m=+3319.272011357" Nov 01 15:13:00.351314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.337909 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" podStartSLOduration=2.337888661 podStartE2EDuration="2.337888661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:13:00.324166759 +0000 UTC m=+3319.382318610" watchObservedRunningTime="2026-11-01 15:13:00.337888661 +0000 UTC m=+3319.396040511" Nov 01 15:13:00.450454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.450421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:00.649902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.649811 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:00.818400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.818365 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Nov 01 15:13:00.818573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:00.818427 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Nov 01 15:13:01.108839 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:13:01.108807 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:13:01.823931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:01.823905 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:13:01.825163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:01.825138 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:13:04.950561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:04.950524 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:08.192453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:08.192421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:09.419203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:09.419174 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:09.607792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:09.607753 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:10.010923 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:10.010884 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:10.309351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:10.309315 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:13:10.309529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:10.309381 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:13:10.478128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:10.478096 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:11.108743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:11.108711 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:13:11.109108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:11.109089 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:13:13.219262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:13.219227 5473 generic.go:356] "Generic (PLEG): container finished" podID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerID="a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514" exitCode=0 Nov 01 15:13:13.219262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:13.219238 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" event={"ID":"6abf39f2-78ae-44ce-801b-dcaab11f5802","Type":"ContainerDied","Data":"a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514"} Nov 01 15:13:13.219714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:13.219631 5473 scope.go:122] "RemoveContainer" containerID="a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514" Nov 01 15:13:13.221390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:13.221364 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:13:13.221464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:13.221431 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:13:13.221507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:13.221465 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:13:14.205978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.205944 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:14.258602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.258559 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" event={"ID":"6abf39f2-78ae-44ce-801b-dcaab11f5802","Type":"ContainerStarted","Data":"f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72"} Nov 01 15:13:14.568123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.568080 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:14.750845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.750801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:14.794913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.794887 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:13:14.795069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.795049 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 01 15:13:14.795122 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.795102 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 01 15:13:14.918578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:14.918499 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:15.795600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:15.795564 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 01 15:13:15.796031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:15.795618 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 01 15:13:16.794716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:16.794680 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 01 15:13:16.794897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:16.794726 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 01 15:13:17.795179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:17.795152 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 01 15:13:17.795565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:17.795196 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 01 15:13:17.958868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:17.958824 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:18.150638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.150546 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:18.323638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.323604 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf"] Nov 01 15:13:18.351497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.351462 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.382656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.382621 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf"] Nov 01 15:13:18.494866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.490191 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.495371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.495331 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnqqf\" (UniqueName: \"kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf\") pod \"payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.499587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.499559 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-c89271\"" Nov 01 15:13:18.499755 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.499738 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-c89271-dockercfg-qxk62\"" Nov 01 15:13:18.600736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.600708 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnqqf\" (UniqueName: \"kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf\") pod \"payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.601040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.601023 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.601705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.601683 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.658784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.658752 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnqqf\" (UniqueName: \"kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf\") pod \"payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.666301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.666271 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:18.795507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.795036 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 01 15:13:18.795507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.795095 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 01 15:13:18.860035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:18.859982 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:19.055027 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:13:19.054995 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2083f500_32e8_40b7_91d0_dfc6f1ddea86.slice/crio-5330882a8fc1bab6bcbdf495309d43ec84a1a1c6a91b5a787f62800b5dde097c WatchSource:0}: Error finding container 5330882a8fc1bab6bcbdf495309d43ec84a1a1c6a91b5a787f62800b5dde097c: Status 404 returned error can't find the container with id 5330882a8fc1bab6bcbdf495309d43ec84a1a1c6a91b5a787f62800b5dde097c Nov 01 15:13:19.057512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.057486 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf"] Nov 01 15:13:19.298997 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.298964 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" event={"ID":"2083f500-32e8-40b7-91d0-dfc6f1ddea86","Type":"ContainerStarted","Data":"246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe"} Nov 01 15:13:19.299237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.299027 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:19.299237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.299050 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" event={"ID":"2083f500-32e8-40b7-91d0-dfc6f1ddea86","Type":"ContainerStarted","Data":"5330882a8fc1bab6bcbdf495309d43ec84a1a1c6a91b5a787f62800b5dde097c"} Nov 01 15:13:19.415341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.415245 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" podStartSLOduration=1.415228817 podStartE2EDuration="1.415228817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:13:19.414432858 +0000 UTC m=+3338.472584707" watchObservedRunningTime="2026-11-01 15:13:19.415228817 +0000 UTC m=+3338.473380666" Nov 01 15:13:19.799604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.799572 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:13:19.800116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.799984 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:13:19.800752 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:19.800730 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:13:23.314476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:23.314401 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:26.974099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:26.974065 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:27.134373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:27.134340 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:27.717095 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:27.717058 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:30.310712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:30.310685 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:13:31.108296 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:13:31.108262 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:13:32.108849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:32.108813 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:36.049261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:36.049183 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w"] Nov 01 15:13:36.099077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:36.099046 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d7e5f4-ts69w"] Nov 01 15:13:36.508126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:36.508049 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:37.083239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:37.083192 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c567c7a4-5dbb-426f-9721-3f084d24b005" path="/var/lib/kubelet/pods/c567c7a4-5dbb-426f-9721-3f084d24b005/volumes" Nov 01 15:13:40.989066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:40.989031 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:45.437856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:45.437820 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:45.706232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:45.706130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:45.923664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:45.923595 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:47.049508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:47.049475 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp"] Nov 01 15:13:47.099260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:47.099232 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef7ec078-kdgrp"] Nov 01 15:13:49.082531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:49.082497 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37c19139-3cec-45bf-9911-586887b31e85" path="/var/lib/kubelet/pods/37c19139-3cec-45bf-9911-586887b31e85/volumes" Nov 01 15:13:49.822381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:49.822345 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:54.215464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:54.215416 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:58.746399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:58.746361 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:13:59.052646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:59.052616 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq"] Nov 01 15:13:59.102576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:13:59.102544 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf9a36d-l2jbq"] Nov 01 15:14:00.941284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:00.941240 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:01.086615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:01.086579 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4" path="/var/lib/kubelet/pods/f323fb7a-8c38-4c15-bd5a-c6e21eacb5a4/volumes" Nov 01 15:14:01.109862 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:14:01.109824 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:14:03.090267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:03.090236 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:03.292749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:03.292709 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:03.455861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:03.455780 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:04.219029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:04.219000 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:07.792279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:07.792248 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:10.055346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:10.055311 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x"] Nov 01 15:14:10.105327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:10.105294 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32ccbb09-rzx7x"] Nov 01 15:14:11.083013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:11.082980 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c19fabe0-0f41-4d1b-98be-544aed6c41c4" path="/var/lib/kubelet/pods/c19fabe0-0f41-4d1b-98be-544aed6c41c4/volumes" Nov 01 15:14:12.231159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:12.231127 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:13.493763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:13.493731 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:13.759844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:13.759758 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:16.082445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:16.082415 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:16.800304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:16.800267 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:19.695351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:19.695317 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:20.934843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:20.934807 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:21.397339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.397304 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr"] Nov 01 15:14:21.401764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.401740 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.479200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.479159 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr"] Nov 01 15:14:21.512648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.512619 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-928a13-openshift-default-dockercfg-m77tj\"" Nov 01 15:14:21.548050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.547967 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548051 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548078 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hd6\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548135 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548316 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548373 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548413 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548458 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.548719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.548499 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.559143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.559092 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:21.649793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.649705 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.649793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.649760 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.649827 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.649871 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.649895 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hd6\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.649960 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650130 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650166 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650181 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650253 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650285 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650303 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650509 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.650961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.650937 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.652628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.652601 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.652890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.652869 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.707777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.707752 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.708267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.708240 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hd6\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6\") pod \"e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.717892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.717867 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:21.764864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.764793 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:21.947836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:21.947755 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:22.138547 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:14:22.138513 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b65ea53_8ec4_4ac8_8063_755d1f64faec.slice/crio-4d1635dde60f7632c12a92ffc19f25da72b3bf678174306d945988af77493cc1 WatchSource:0}: Error finding container 4d1635dde60f7632c12a92ffc19f25da72b3bf678174306d945988af77493cc1: Status 404 returned error can't find the container with id 4d1635dde60f7632c12a92ffc19f25da72b3bf678174306d945988af77493cc1 Nov 01 15:14:22.141492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.141459 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:14:22.141584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.141527 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:14:22.141584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.141556 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:14:22.141838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.141818 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr"] Nov 01 15:14:22.768538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.768496 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" event={"ID":"6b65ea53-8ec4-4ac8-8063-755d1f64faec","Type":"ContainerStarted","Data":"514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910"} Nov 01 15:14:22.768749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.768552 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" event={"ID":"6b65ea53-8ec4-4ac8-8063-755d1f64faec","Type":"ContainerStarted","Data":"4d1635dde60f7632c12a92ffc19f25da72b3bf678174306d945988af77493cc1"} Nov 01 15:14:22.924606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:22.924554 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podStartSLOduration=1.9245410459999999 podStartE2EDuration="1.924541046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:14:22.920626008 +0000 UTC m=+3401.978777857" watchObservedRunningTime="2026-11-01 15:14:22.924541046 +0000 UTC m=+3401.982692919" Nov 01 15:14:23.718856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:23.718823 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:23.724515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:23.724492 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:23.775814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:23.775787 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:23.776641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:23.776622 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:14:25.648861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:25.648828 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:25.850742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:25.850692 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:26.017963 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:26.017926 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:26.676045 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:26.676011 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:29.647241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:29.646876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:29.817773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:29.817716 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:31.122098 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:14:31.122059 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:14:31.714497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:31.714458 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:33.852756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:33.852726 5473 generic.go:356] "Generic (PLEG): container finished" podID="d75df714-1cda-42e8-a61f-da2de949f015" containerID="caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4" exitCode=0 Nov 01 15:14:33.853145 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:33.852799 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" event={"ID":"d75df714-1cda-42e8-a61f-da2de949f015","Type":"ContainerDied","Data":"caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4"} Nov 01 15:14:33.853307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:33.853291 5473 scope.go:122] "RemoveContainer" containerID="caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4" Nov 01 15:14:33.855130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:33.855101 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:14:33.855264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:33.855172 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:14:33.855264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:33.855206 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:14:34.864833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:34.864784 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" event={"ID":"d75df714-1cda-42e8-a61f-da2de949f015","Type":"ContainerStarted","Data":"890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907"} Nov 01 15:14:35.121735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:35.121647 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:14:35.122605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:35.122580 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 01 15:14:35.122685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:35.122630 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 01 15:14:36.121651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:36.121615 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 01 15:14:36.122058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:36.121681 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 01 15:14:36.313076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:36.313039 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:37.121729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:37.121695 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 01 15:14:37.122121 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:37.121744 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 01 15:14:38.121716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:38.121677 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 01 15:14:38.121927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:38.121744 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 01 15:14:39.120933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:39.120900 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Nov 01 15:14:39.121150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:39.120956 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Nov 01 15:14:39.932799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:39.932762 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:14:40.126250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:40.126196 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:14:40.735703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:40.735662 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:40.912121 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:40.912094 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:14:43.699133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:43.699100 5473 scope.go:122] "RemoveContainer" containerID="ccd09a466e8f08164581cad3690f2d528e0bee52dc54cc3de8c4f43832e1de14" Nov 01 15:14:43.709880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:43.709861 5473 scope.go:122] "RemoveContainer" containerID="0c807e71df9dd772fa5cccd69c1ca04ce2b7301d152ed027f08991a80b9ab7ad" Nov 01 15:14:45.117940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:45.117907 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:45.294237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:45.294185 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:45.457703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:45.457621 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:49.671642 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:49.671608 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:54.214674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:54.214590 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:14:58.737921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:14:58.737876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:00.161897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.161868 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q"] Nov 01 15:15:00.167771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.167751 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv"] Nov 01 15:15:00.167890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.167868 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.172316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.172296 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z"] Nov 01 15:15:00.172424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.172391 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.176502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.176482 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.210865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.210845 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws"] Nov 01 15:15:00.214939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.214921 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj"] Nov 01 15:15:00.215041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.215021 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.220512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.220494 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp"] Nov 01 15:15:00.220621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.220604 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.224534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.224515 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9"] Nov 01 15:15:00.224635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.224616 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.228565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.228549 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd"] Nov 01 15:15:00.228683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.228665 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.233840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.233818 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp"] Nov 01 15:15:00.233949 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.233909 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.237963 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.237947 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc"] Nov 01 15:15:00.238069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.238052 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.242588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.242566 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.257785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.257760 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:15:00.261730 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.261708 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58"] Nov 01 15:15:00.266070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.266052 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd"] Nov 01 15:15:00.266195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.266175 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.283670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.283651 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r"] Nov 01 15:15:00.283798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.283777 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.291872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.291848 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92"] Nov 01 15:15:00.292041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.292022 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.297112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.297092 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q"] Nov 01 15:15:00.297255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.297133 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z"] Nov 01 15:15:00.297255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.297227 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.312288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.312265 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp"] Nov 01 15:15:00.316388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.316363 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws"] Nov 01 15:15:00.319652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319621 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.319746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319674 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.319805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319745 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxt77\" (UniqueName: \"kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.319805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319780 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twvqf\" (UniqueName: \"kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf\") pod \"maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.319908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319828 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhxm\" (UniqueName: \"kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm\") pod \"maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.319908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319862 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxmk\" (UniqueName: \"kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.319978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.319925 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.320028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320001 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5z4\" (UniqueName: \"kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.320105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320086 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzdcl\" (UniqueName: \"kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.320158 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320124 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm874\" (UniqueName: \"kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.320195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320160 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.320319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320303 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.320391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320347 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.320391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320374 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42vx\" (UniqueName: \"kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx\") pod \"maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.320536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320517 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gk5\" (UniqueName: \"kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.320602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320554 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd"] Nov 01 15:15:00.320682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320616 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf472\" (UniqueName: \"kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.320740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320707 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.320798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320738 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.320798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320777 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.320912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.320806 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.324298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.324236 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9"] Nov 01 15:15:00.366399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.366369 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc"] Nov 01 15:15:00.401147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.401118 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92"] Nov 01 15:15:00.422100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422021 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzdcl\" (UniqueName: \"kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.422100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422055 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm874\" (UniqueName: \"kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.422100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422079 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8j4v\" (UniqueName: \"kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.422416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422114 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.422416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422310 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnn6w\" (UniqueName: \"kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.422416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422358 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.422596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422428 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.422596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422463 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42vx\" (UniqueName: \"kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx\") pod \"maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.422596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422512 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gk5\" (UniqueName: \"kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.422596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422541 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf472\" (UniqueName: \"kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.422596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422570 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422599 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422640 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422669 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422698 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422742 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422811 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.422875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422847 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.423291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422928 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxt77\" (UniqueName: \"kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.423291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422933 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.423291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422948 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.423291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.422966 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twvqf\" (UniqueName: \"kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf\") pod \"maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.423511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.423485 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.423734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.423709 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.424746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.423895 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.424853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424135 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.424853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424170 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.424853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424795 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhxm\" (UniqueName: \"kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm\") pod \"maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.424853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424710 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.424853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424720 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.424853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424710 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.425179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424861 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwkt\" (UniqueName: \"kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.425179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424908 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxmk\" (UniqueName: \"kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.425179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.424995 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8dd\" (UniqueName: \"kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.425179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.425027 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.425179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.425123 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.425179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.425155 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5z4\" (UniqueName: \"kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.425593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.425578 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.439034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.438710 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58"] Nov 01 15:15:00.453743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.444356 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv"] Nov 01 15:15:00.453743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.449918 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r"] Nov 01 15:15:00.454918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.454895 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj"] Nov 01 15:15:00.458482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.458460 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp"] Nov 01 15:15:00.461574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.461556 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd"] Nov 01 15:15:00.482474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.482447 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gk5\" (UniqueName: \"kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.482738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.482718 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm874\" (UniqueName: \"kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874\") pod \"maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.482853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.482826 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzdcl\" (UniqueName: \"kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.484708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.484687 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twvqf\" (UniqueName: \"kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf\") pod \"maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.484708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.484698 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxt77\" (UniqueName: \"kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.488884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.488863 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf472\" (UniqueName: \"kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.488987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.488961 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxmk\" (UniqueName: \"kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.489060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.489001 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhxm\" (UniqueName: \"kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm\") pod \"maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.489190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.489171 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5z4\" (UniqueName: \"kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.490616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.490597 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42vx\" (UniqueName: \"kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx\") pod \"maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.491830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.491815 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:00.499361 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.499340 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:00.526734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.526699 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.526881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.526757 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwkt\" (UniqueName: \"kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.526881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.526812 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8dd\" (UniqueName: \"kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.526998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.526904 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.526998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.526987 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8j4v\" (UniqueName: \"kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.527108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.527029 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnn6w\" (UniqueName: \"kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.527167 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.527106 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.527167 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.527133 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.528055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.528027 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.529189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.529166 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.530489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.530405 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.530578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.530519 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:00.530928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.530905 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.540132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.540105 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:00.550966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.550938 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:00.557725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.557700 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:00.565534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.565455 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:00.581304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.580148 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:00.592975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.592917 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8dd\" (UniqueName: \"kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd\") pod \"maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.594505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.594335 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:00.594799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.594774 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwkt\" (UniqueName: \"kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt\") pod \"maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.596510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.596466 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8j4v\" (UniqueName: \"kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.597327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.597283 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnn6w\" (UniqueName: \"kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.607266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.607105 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:00.620471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.619428 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:00.628598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.628575 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:00.647698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.646608 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:00.788324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.788174 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:00.893507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:00.892748 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv"] Nov 01 15:15:01.062159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.062122 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerStarted","Data":"127b4a49a60d4f49a57d2523d74967d3f0f1f635135fa435e01dd5439c084771"} Nov 01 15:15:01.076293 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.076257 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731d4b63_f3ad_468a_a224_576f728e9e06.slice/crio-0eb477dd6461fbb41f72ba800bc6e63991a932d57548747691c41dfe470dbf11 WatchSource:0}: Error finding container 0eb477dd6461fbb41f72ba800bc6e63991a932d57548747691c41dfe470dbf11: Status 404 returned error can't find the container with id 0eb477dd6461fbb41f72ba800bc6e63991a932d57548747691c41dfe470dbf11 Nov 01 15:15:01.091847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.091821 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z"] Nov 01 15:15:01.122189 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:01.122156 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:15:01.160977 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.160949 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf4cfa9_eae6_43fc_9457_440c5940b693.slice/crio-9d1806c82230d91c4dafca10e79f9b628dabc967662f42d33ef3fc2a400c0742 WatchSource:0}: Error finding container 9d1806c82230d91c4dafca10e79f9b628dabc967662f42d33ef3fc2a400c0742: Status 404 returned error can't find the container with id 9d1806c82230d91c4dafca10e79f9b628dabc967662f42d33ef3fc2a400c0742 Nov 01 15:15:01.165082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.165059 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws"] Nov 01 15:15:01.188047 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.188023 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874f4254_ae13_4ec0_8afb_c4342ca765ed.slice/crio-f34194f263a5c8da7ea3d4a81c9b86ec16d8ab699505025c86edab9817b16770 WatchSource:0}: Error finding container f34194f263a5c8da7ea3d4a81c9b86ec16d8ab699505025c86edab9817b16770: Status 404 returned error can't find the container with id f34194f263a5c8da7ea3d4a81c9b86ec16d8ab699505025c86edab9817b16770 Nov 01 15:15:01.193005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.192982 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj"] Nov 01 15:15:01.215264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.215241 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp"] Nov 01 15:15:01.220391 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.220368 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8780c7c8_c128_4639_bcc0_7d806cbf5fb9.slice/crio-ea204c93d4a48c37683b48567ca833b5dd3990c20c9d8ecf0f2c6deb69016a6e WatchSource:0}: Error finding container ea204c93d4a48c37683b48567ca833b5dd3990c20c9d8ecf0f2c6deb69016a6e: Status 404 returned error can't find the container with id ea204c93d4a48c37683b48567ca833b5dd3990c20c9d8ecf0f2c6deb69016a6e Nov 01 15:15:01.234477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.234452 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp"] Nov 01 15:15:01.237139 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.237114 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed0f6ce_0634_426b_a25a_b8dae42bb8d8.slice/crio-b4c8ede60acb58ae9493a0b408f398fe654754ce62ff13fa1d7c45633c081ab8 WatchSource:0}: Error finding container b4c8ede60acb58ae9493a0b408f398fe654754ce62ff13fa1d7c45633c081ab8: Status 404 returned error can't find the container with id b4c8ede60acb58ae9493a0b408f398fe654754ce62ff13fa1d7c45633c081ab8 Nov 01 15:15:01.251387 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.251365 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a7a920_701a_4c4f_bc06_bb73f00573e1.slice/crio-b72216238bd3dff2e8b9d2000f082a72b4e5d9fbe8c34f674c9bc25c45404541 WatchSource:0}: Error finding container b72216238bd3dff2e8b9d2000f082a72b4e5d9fbe8c34f674c9bc25c45404541: Status 404 returned error can't find the container with id b72216238bd3dff2e8b9d2000f082a72b4e5d9fbe8c34f674c9bc25c45404541 Nov 01 15:15:01.255769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.255747 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9"] Nov 01 15:15:01.257364 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.257339 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebdb2c6_1b1f_4b97_b658_8f806387b52c.slice/crio-51515ec7522fee927b0acac6a07615f38f8fa8ed9068a6719d18306246df280e WatchSource:0}: Error finding container 51515ec7522fee927b0acac6a07615f38f8fa8ed9068a6719d18306246df280e: Status 404 returned error can't find the container with id 51515ec7522fee927b0acac6a07615f38f8fa8ed9068a6719d18306246df280e Nov 01 15:15:01.259508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.259484 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:01.261690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.261663 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q"] Nov 01 15:15:01.313153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.313126 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd"] Nov 01 15:15:01.332930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.332905 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc"] Nov 01 15:15:01.360585 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.360564 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58"] Nov 01 15:15:01.373073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.373051 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r"] Nov 01 15:15:01.396927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.396906 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92"] Nov 01 15:15:01.433376 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.433317 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24545ec6_f2fa_4658_9287_e915c228e8e0.slice/crio-2af950df38a31a720464bd30077d3954647f706a156a52c3dbfa1d399a76779f WatchSource:0}: Error finding container 2af950df38a31a720464bd30077d3954647f706a156a52c3dbfa1d399a76779f: Status 404 returned error can't find the container with id 2af950df38a31a720464bd30077d3954647f706a156a52c3dbfa1d399a76779f Nov 01 15:15:01.436438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:01.436414 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd"] Nov 01 15:15:01.439147 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.439125 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57a32dc_2a05_4f72_8a8c_aeb14c9ab07b.slice/crio-814ba5b4bd5e6d9274f45ee7c50c042251a11109834f216153db75667f54b716 WatchSource:0}: Error finding container 814ba5b4bd5e6d9274f45ee7c50c042251a11109834f216153db75667f54b716: Status 404 returned error can't find the container with id 814ba5b4bd5e6d9274f45ee7c50c042251a11109834f216153db75667f54b716 Nov 01 15:15:01.542691 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.541443 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb9ab947_ed5d_4f22_baf4_555f4da6fe44.slice/crio-100e31594ea95af04d84b8b9139194ff67616f4bfd421f094d89ac2ac0a4607e WatchSource:0}: Error finding container 100e31594ea95af04d84b8b9139194ff67616f4bfd421f094d89ac2ac0a4607e: Status 404 returned error can't find the container with id 100e31594ea95af04d84b8b9139194ff67616f4bfd421f094d89ac2ac0a4607e Nov 01 15:15:01.542691 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:01.541826 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02170a0e_d4fc_4024_b1a8_c5d8a4ce524f.slice/crio-a243caa5cc9dedc810c6223cd7cecede0b22ebac7ef8388d92f08544081652fd WatchSource:0}: Error finding container a243caa5cc9dedc810c6223cd7cecede0b22ebac7ef8388d92f08544081652fd: Status 404 returned error can't find the container with id a243caa5cc9dedc810c6223cd7cecede0b22ebac7ef8388d92f08544081652fd Nov 01 15:15:02.072687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.072645 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerStarted","Data":"7f70ee93dc919e93de18e0a400a4d10475c80a84a5461de54f239690e7e0ddc1"} Nov 01 15:15:02.072879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.072712 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerStarted","Data":"9d1806c82230d91c4dafca10e79f9b628dabc967662f42d33ef3fc2a400c0742"} Nov 01 15:15:02.074112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.074083 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" event={"ID":"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b","Type":"ContainerStarted","Data":"814ba5b4bd5e6d9274f45ee7c50c042251a11109834f216153db75667f54b716"} Nov 01 15:15:02.075516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.075489 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" event={"ID":"bb9ab947-ed5d-4f22-baf4-555f4da6fe44","Type":"ContainerStarted","Data":"100e31594ea95af04d84b8b9139194ff67616f4bfd421f094d89ac2ac0a4607e"} Nov 01 15:15:02.076720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.076697 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerStarted","Data":"b4c8ede60acb58ae9493a0b408f398fe654754ce62ff13fa1d7c45633c081ab8"} Nov 01 15:15:02.077960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.077938 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerStarted","Data":"ea204c93d4a48c37683b48567ca833b5dd3990c20c9d8ecf0f2c6deb69016a6e"} Nov 01 15:15:02.079141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.079121 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" event={"ID":"24545ec6-f2fa-4658-9287-e915c228e8e0","Type":"ContainerStarted","Data":"2af950df38a31a720464bd30077d3954647f706a156a52c3dbfa1d399a76779f"} Nov 01 15:15:02.080669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.080648 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerStarted","Data":"c3f7820c2957c471a63595d6d20672477a19be0fc685a4dfc2c9a371407e0174"} Nov 01 15:15:02.080760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.080685 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerStarted","Data":"51515ec7522fee927b0acac6a07615f38f8fa8ed9068a6719d18306246df280e"} Nov 01 15:15:02.081921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.081896 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerStarted","Data":"ecc99acc7a007c52b4a3b09854147acd224cc22b3e837f12606b1f8170b186b1"} Nov 01 15:15:02.083601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.083577 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerStarted","Data":"a7a9e3a5f1e1807fb08cf7b3c9bfa6d4a7deac2dfbb3e38eec2048950755042a"} Nov 01 15:15:02.083704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.083616 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerStarted","Data":"f34194f263a5c8da7ea3d4a81c9b86ec16d8ab699505025c86edab9817b16770"} Nov 01 15:15:02.085126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.085108 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerStarted","Data":"aac4b2fb724b5fdee72963a998c46f8636bb43b4d0a9407b345e75b4eb66b5be"} Nov 01 15:15:02.086411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.086391 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" event={"ID":"84b153d4-a17d-4cd2-ab3a-308be9b516f9","Type":"ContainerStarted","Data":"a18d3c02f31c172f53582b91585a41037be0493f1bf4127dfd57308463a8be99"} Nov 01 15:15:02.087962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.087935 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerStarted","Data":"24a9f091b48e15392e33a83d52be77c4b63c96712d957957a84f355a7c6c32ea"} Nov 01 15:15:02.088079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.087968 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerStarted","Data":"0eb477dd6461fbb41f72ba800bc6e63991a932d57548747691c41dfe470dbf11"} Nov 01 15:15:02.089265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.089243 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" event={"ID":"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f","Type":"ContainerStarted","Data":"a243caa5cc9dedc810c6223cd7cecede0b22ebac7ef8388d92f08544081652fd"} Nov 01 15:15:02.090659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.090637 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerStarted","Data":"e64ce951590ebdaa5043ed66377691b75f6b567463c2f2a6edaa238ed70f1e34"} Nov 01 15:15:02.090741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.090673 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerStarted","Data":"b72216238bd3dff2e8b9d2000f082a72b4e5d9fbe8c34f674c9bc25c45404541"} Nov 01 15:15:02.193413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.193352 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" podStartSLOduration=2.193333939 podStartE2EDuration="2.193333939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.189779378 +0000 UTC m=+3441.247931227" watchObservedRunningTime="2026-11-01 15:15:02.193333939 +0000 UTC m=+3441.251485790" Nov 01 15:15:02.303545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.303451 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" podStartSLOduration=2.303435611 podStartE2EDuration="2.303435611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.30211123 +0000 UTC m=+3441.360263079" watchObservedRunningTime="2026-11-01 15:15:02.303435611 +0000 UTC m=+3441.361587463" Nov 01 15:15:02.416023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.415967 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" podStartSLOduration=2.415953125 podStartE2EDuration="2.415953125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.415374554 +0000 UTC m=+3441.473526403" watchObservedRunningTime="2026-11-01 15:15:02.415953125 +0000 UTC m=+3441.474104971" Nov 01 15:15:02.534875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.534825 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" podStartSLOduration=2.534811816 podStartE2EDuration="2.534811816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.52824279 +0000 UTC m=+3441.586394642" watchObservedRunningTime="2026-11-01 15:15:02.534811816 +0000 UTC m=+3441.592963686" Nov 01 15:15:02.642151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.642104 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" podStartSLOduration=2.642091573 podStartE2EDuration="2.642091573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.641517563 +0000 UTC m=+3441.699669424" watchObservedRunningTime="2026-11-01 15:15:02.642091573 +0000 UTC m=+3441.700243422" Nov 01 15:15:02.757353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:02.757300 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" podStartSLOduration=2.75728407 podStartE2EDuration="2.75728407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:02.754420108 +0000 UTC m=+3441.812571959" watchObservedRunningTime="2026-11-01 15:15:02.75728407 +0000 UTC m=+3441.815435923" Nov 01 15:15:03.100134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.100054 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerStarted","Data":"f71ef3941c09b0f1a19f544147e335d63879642d788935644e459c39b2f4093d"} Nov 01 15:15:03.101958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.101927 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerStarted","Data":"0b291947728b48c882238a60cea280fbaf23a48fa559db00a235c737c68757f5"} Nov 01 15:15:03.103779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.103752 5473 generic.go:356] "Generic (PLEG): container finished" podID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerID="78cb115ca19d7f1b9c62b437231ca459eb644a8b5cf66d779c4c6e919de15c99" exitCode=60 Nov 01 15:15:03.103882 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.103801 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" event={"ID":"24545ec6-f2fa-4658-9287-e915c228e8e0","Type":"ContainerDied","Data":"78cb115ca19d7f1b9c62b437231ca459eb644a8b5cf66d779c4c6e919de15c99"} Nov 01 15:15:03.104062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.104048 5473 scope.go:122] "RemoveContainer" containerID="78cb115ca19d7f1b9c62b437231ca459eb644a8b5cf66d779c4c6e919de15c99" Nov 01 15:15:03.105563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.105545 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerID="c3f7820c2957c471a63595d6d20672477a19be0fc685a4dfc2c9a371407e0174" exitCode=60 Nov 01 15:15:03.105643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.105594 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerDied","Data":"c3f7820c2957c471a63595d6d20672477a19be0fc685a4dfc2c9a371407e0174"} Nov 01 15:15:03.105800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.105786 5473 scope.go:122] "RemoveContainer" containerID="c3f7820c2957c471a63595d6d20672477a19be0fc685a4dfc2c9a371407e0174" Nov 01 15:15:03.116511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.114907 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerStarted","Data":"2c30a5f044b9c9023bb32db9af97a34365bd58ff53bcb8cf8869a342f96d0c31"} Nov 01 15:15:03.119950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.119923 5473 generic.go:356] "Generic (PLEG): container finished" podID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerID="a7a9e3a5f1e1807fb08cf7b3c9bfa6d4a7deac2dfbb3e38eec2048950755042a" exitCode=60 Nov 01 15:15:03.120074 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.120000 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerDied","Data":"a7a9e3a5f1e1807fb08cf7b3c9bfa6d4a7deac2dfbb3e38eec2048950755042a"} Nov 01 15:15:03.120210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.120195 5473 scope.go:122] "RemoveContainer" containerID="a7a9e3a5f1e1807fb08cf7b3c9bfa6d4a7deac2dfbb3e38eec2048950755042a" Nov 01 15:15:03.121785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.121763 5473 generic.go:356] "Generic (PLEG): container finished" podID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerID="aac4b2fb724b5fdee72963a998c46f8636bb43b4d0a9407b345e75b4eb66b5be" exitCode=60 Nov 01 15:15:03.121880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.121847 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerDied","Data":"aac4b2fb724b5fdee72963a998c46f8636bb43b4d0a9407b345e75b4eb66b5be"} Nov 01 15:15:03.122029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.122009 5473 scope.go:122] "RemoveContainer" containerID="aac4b2fb724b5fdee72963a998c46f8636bb43b4d0a9407b345e75b4eb66b5be" Nov 01 15:15:03.123448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.123429 5473 generic.go:356] "Generic (PLEG): container finished" podID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerID="d80e510ee018fe2977908d7c08eeded29f1600343d7ab97682201541ebe3ac24" exitCode=60 Nov 01 15:15:03.123549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.123508 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" event={"ID":"84b153d4-a17d-4cd2-ab3a-308be9b516f9","Type":"ContainerDied","Data":"d80e510ee018fe2977908d7c08eeded29f1600343d7ab97682201541ebe3ac24"} Nov 01 15:15:03.123718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.123703 5473 scope.go:122] "RemoveContainer" containerID="d80e510ee018fe2977908d7c08eeded29f1600343d7ab97682201541ebe3ac24" Nov 01 15:15:03.127157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.127134 5473 generic.go:356] "Generic (PLEG): container finished" podID="731d4b63-f3ad-468a-a224-576f728e9e06" containerID="24a9f091b48e15392e33a83d52be77c4b63c96712d957957a84f355a7c6c32ea" exitCode=60 Nov 01 15:15:03.127295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.127226 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerDied","Data":"24a9f091b48e15392e33a83d52be77c4b63c96712d957957a84f355a7c6c32ea"} Nov 01 15:15:03.127522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.127504 5473 scope.go:122] "RemoveContainer" containerID="24a9f091b48e15392e33a83d52be77c4b63c96712d957957a84f355a7c6c32ea" Nov 01 15:15:03.129609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.129588 5473 generic.go:356] "Generic (PLEG): container finished" podID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerID="872614f06a7a1acf27bcd464a90554161c28009bdd315ce980e6d47f5889af97" exitCode=60 Nov 01 15:15:03.129785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.129759 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" event={"ID":"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f","Type":"ContainerDied","Data":"872614f06a7a1acf27bcd464a90554161c28009bdd315ce980e6d47f5889af97"} Nov 01 15:15:03.130031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.130012 5473 scope.go:122] "RemoveContainer" containerID="872614f06a7a1acf27bcd464a90554161c28009bdd315ce980e6d47f5889af97" Nov 01 15:15:03.132043 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.132020 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerID="e64ce951590ebdaa5043ed66377691b75f6b567463c2f2a6edaa238ed70f1e34" exitCode=60 Nov 01 15:15:03.132260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.132177 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerDied","Data":"e64ce951590ebdaa5043ed66377691b75f6b567463c2f2a6edaa238ed70f1e34"} Nov 01 15:15:03.132424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.132399 5473 scope.go:122] "RemoveContainer" containerID="e64ce951590ebdaa5043ed66377691b75f6b567463c2f2a6edaa238ed70f1e34" Nov 01 15:15:03.134567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.134547 5473 generic.go:356] "Generic (PLEG): container finished" podID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerID="7f70ee93dc919e93de18e0a400a4d10475c80a84a5461de54f239690e7e0ddc1" exitCode=60 Nov 01 15:15:03.134646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.134631 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerDied","Data":"7f70ee93dc919e93de18e0a400a4d10475c80a84a5461de54f239690e7e0ddc1"} Nov 01 15:15:03.134987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.134966 5473 scope.go:122] "RemoveContainer" containerID="7f70ee93dc919e93de18e0a400a4d10475c80a84a5461de54f239690e7e0ddc1" Nov 01 15:15:03.144637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.144609 5473 generic.go:356] "Generic (PLEG): container finished" podID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerID="6ea70b3ce1d9ecb3ad8f382c816604ceac6cb249a8134ea3a24ac3a03fc5024d" exitCode=60 Nov 01 15:15:03.144748 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.144724 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" event={"ID":"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b","Type":"ContainerDied","Data":"6ea70b3ce1d9ecb3ad8f382c816604ceac6cb249a8134ea3a24ac3a03fc5024d"} Nov 01 15:15:03.145019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.144972 5473 scope.go:122] "RemoveContainer" containerID="6ea70b3ce1d9ecb3ad8f382c816604ceac6cb249a8134ea3a24ac3a03fc5024d" Nov 01 15:15:03.147126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.147020 5473 generic.go:356] "Generic (PLEG): container finished" podID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerID="ce96b9decc6e3664d983d0087d78bd58392edbae4588b2b27327cec14c099de4" exitCode=60 Nov 01 15:15:03.147126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.147104 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" event={"ID":"bb9ab947-ed5d-4f22-baf4-555f4da6fe44","Type":"ContainerDied","Data":"ce96b9decc6e3664d983d0087d78bd58392edbae4588b2b27327cec14c099de4"} Nov 01 15:15:03.148307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.147572 5473 scope.go:122] "RemoveContainer" containerID="ce96b9decc6e3664d983d0087d78bd58392edbae4588b2b27327cec14c099de4" Nov 01 15:15:03.184280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.184244 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:03.217733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.217665 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" podStartSLOduration=3.217644552 podStartE2EDuration="3.217644552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:03.212583418 +0000 UTC m=+3442.270735268" watchObservedRunningTime="2026-11-01 15:15:03.217644552 +0000 UTC m=+3442.275796403" Nov 01 15:15:03.665614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:03.665567 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" podStartSLOduration=3.665553474 podStartE2EDuration="3.665553474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:03.662853015 +0000 UTC m=+3442.721004864" watchObservedRunningTime="2026-11-01 15:15:03.665553474 +0000 UTC m=+3442.723705324" Nov 01 15:15:04.157098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.157067 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerID="f71ef3941c09b0f1a19f544147e335d63879642d788935644e459c39b2f4093d" exitCode=60 Nov 01 15:15:04.157291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.157114 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerDied","Data":"f71ef3941c09b0f1a19f544147e335d63879642d788935644e459c39b2f4093d"} Nov 01 15:15:04.157524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.157506 5473 scope.go:122] "RemoveContainer" containerID="f71ef3941c09b0f1a19f544147e335d63879642d788935644e459c39b2f4093d" Nov 01 15:15:04.158966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.158946 5473 generic.go:356] "Generic (PLEG): container finished" podID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerID="0b291947728b48c882238a60cea280fbaf23a48fa559db00a235c737c68757f5" exitCode=60 Nov 01 15:15:04.159086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.159031 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerDied","Data":"0b291947728b48c882238a60cea280fbaf23a48fa559db00a235c737c68757f5"} Nov 01 15:15:04.159377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.159288 5473 scope.go:122] "RemoveContainer" containerID="0b291947728b48c882238a60cea280fbaf23a48fa559db00a235c737c68757f5" Nov 01 15:15:04.161449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.161329 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" event={"ID":"24545ec6-f2fa-4658-9287-e915c228e8e0","Type":"ContainerStarted","Data":"741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d"} Nov 01 15:15:04.163210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.163189 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerStarted","Data":"93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617"} Nov 01 15:15:04.165051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.165028 5473 generic.go:356] "Generic (PLEG): container finished" podID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerID="2c30a5f044b9c9023bb32db9af97a34365bd58ff53bcb8cf8869a342f96d0c31" exitCode=60 Nov 01 15:15:04.165159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.165092 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerDied","Data":"2c30a5f044b9c9023bb32db9af97a34365bd58ff53bcb8cf8869a342f96d0c31"} Nov 01 15:15:04.165376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.165356 5473 scope.go:122] "RemoveContainer" containerID="2c30a5f044b9c9023bb32db9af97a34365bd58ff53bcb8cf8869a342f96d0c31" Nov 01 15:15:04.166963 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.166940 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerStarted","Data":"4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06"} Nov 01 15:15:04.168868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.168844 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerStarted","Data":"f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae"} Nov 01 15:15:04.170923 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.170902 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" event={"ID":"84b153d4-a17d-4cd2-ab3a-308be9b516f9","Type":"ContainerStarted","Data":"94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f"} Nov 01 15:15:04.172920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.172882 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerStarted","Data":"c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818"} Nov 01 15:15:04.175146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.175092 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" event={"ID":"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f","Type":"ContainerStarted","Data":"4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f"} Nov 01 15:15:04.177210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.177191 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerStarted","Data":"aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc"} Nov 01 15:15:04.182744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.182312 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerStarted","Data":"d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9"} Nov 01 15:15:04.184993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.184968 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" event={"ID":"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b","Type":"ContainerStarted","Data":"7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5"} Nov 01 15:15:04.187013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.186989 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" event={"ID":"bb9ab947-ed5d-4f22-baf4-555f4da6fe44","Type":"ContainerStarted","Data":"420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1"} Nov 01 15:15:04.352057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.351230 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" podStartSLOduration=4.351202157 podStartE2EDuration="4.351202157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:04.348596092 +0000 UTC m=+3443.406747943" watchObservedRunningTime="2026-11-01 15:15:04.351202157 +0000 UTC m=+3443.409354006" Nov 01 15:15:04.819043 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:04.818994 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" podStartSLOduration=4.8189810269999995 podStartE2EDuration="4.818981027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:04.815342461 +0000 UTC m=+3443.873494322" watchObservedRunningTime="2026-11-01 15:15:04.818981027 +0000 UTC m=+3443.877132877" Nov 01 15:15:05.045417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.045369 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" podStartSLOduration=5.045356679 podStartE2EDuration="5.045356679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:05.042014037 +0000 UTC m=+3444.100165886" watchObservedRunningTime="2026-11-01 15:15:05.045356679 +0000 UTC m=+3444.103508528" Nov 01 15:15:05.197090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.197058 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerID="93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617" exitCode=60 Nov 01 15:15:05.197269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.197138 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerDied","Data":"93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617"} Nov 01 15:15:05.197269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.197207 5473 scope.go:122] "RemoveContainer" containerID="c3f7820c2957c471a63595d6d20672477a19be0fc685a4dfc2c9a371407e0174" Nov 01 15:15:05.197482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.197462 5473 scope.go:122] "RemoveContainer" containerID="93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617" Nov 01 15:15:05.197862 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.197838 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q_odh-ai-gateway-infra(0ebdb2c6-1b1f-4b97-b658-8f806387b52c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" Nov 01 15:15:05.199132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.199103 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerStarted","Data":"cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098"} Nov 01 15:15:05.201092 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.201073 5473 generic.go:356] "Generic (PLEG): container finished" podID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerID="4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06" exitCode=60 Nov 01 15:15:05.201211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.201153 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerDied","Data":"4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06"} Nov 01 15:15:05.201373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.201337 5473 scope.go:122] "RemoveContainer" containerID="4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06" Nov 01 15:15:05.201606 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.201583 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj_odh-ai-gateway-infra(874f4254-ae13-4ec0-8afb-c4342ca765ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" Nov 01 15:15:05.203225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.203193 5473 generic.go:356] "Generic (PLEG): container finished" podID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerID="f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae" exitCode=60 Nov 01 15:15:05.203314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.203271 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerDied","Data":"f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae"} Nov 01 15:15:05.203461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.203448 5473 scope.go:122] "RemoveContainer" containerID="f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae" Nov 01 15:15:05.203660 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.203637 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv_odh-ai-gateway-infra(1538b70d-4c8e-4226-afc6-5a4bcaa6836d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" Nov 01 15:15:05.205372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.205347 5473 generic.go:356] "Generic (PLEG): container finished" podID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerID="94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f" exitCode=60 Nov 01 15:15:05.205463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.205408 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" event={"ID":"84b153d4-a17d-4cd2-ab3a-308be9b516f9","Type":"ContainerDied","Data":"94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f"} Nov 01 15:15:05.205714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.205697 5473 scope.go:122] "RemoveContainer" containerID="94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f" Nov 01 15:15:05.205948 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.205926 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58_odh-ai-gateway-infra(84b153d4-a17d-4cd2-ab3a-308be9b516f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" Nov 01 15:15:05.207479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.207459 5473 generic.go:356] "Generic (PLEG): container finished" podID="731d4b63-f3ad-468a-a224-576f728e9e06" containerID="c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818" exitCode=60 Nov 01 15:15:05.207567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.207518 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerDied","Data":"c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818"} Nov 01 15:15:05.207735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.207719 5473 scope.go:122] "RemoveContainer" containerID="c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818" Nov 01 15:15:05.207974 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.207956 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z_odh-ai-gateway-infra(731d4b63-f3ad-468a-a224-576f728e9e06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" Nov 01 15:15:05.209526 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.209497 5473 generic.go:356] "Generic (PLEG): container finished" podID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerID="4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f" exitCode=60 Nov 01 15:15:05.209610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.209560 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" event={"ID":"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f","Type":"ContainerDied","Data":"4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f"} Nov 01 15:15:05.209785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.209772 5473 scope.go:122] "RemoveContainer" containerID="4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f" Nov 01 15:15:05.210023 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.209999 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd_odh-ai-gateway-infra(02170a0e-d4fc-4024-b1a8-c5d8a4ce524f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" Nov 01 15:15:05.211565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.211546 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerID="aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc" exitCode=60 Nov 01 15:15:05.211656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.211622 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerDied","Data":"aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc"} Nov 01 15:15:05.211800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.211785 5473 scope.go:122] "RemoveContainer" containerID="aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc" Nov 01 15:15:05.211996 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.211969 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9_odh-ai-gateway-infra(b2a7a920-701a-4c4f-bc06-bb73f00573e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" Nov 01 15:15:05.212485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.212470 5473 scope.go:122] "RemoveContainer" containerID="a7a9e3a5f1e1807fb08cf7b3c9bfa6d4a7deac2dfbb3e38eec2048950755042a" Nov 01 15:15:05.213592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.213572 5473 generic.go:356] "Generic (PLEG): container finished" podID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerID="d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9" exitCode=60 Nov 01 15:15:05.213670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.213625 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerDied","Data":"d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9"} Nov 01 15:15:05.213824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.213810 5473 scope.go:122] "RemoveContainer" containerID="d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9" Nov 01 15:15:05.214030 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.214010 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws_odh-ai-gateway-infra(0cf4cfa9-eae6-43fc-9457-440c5940b693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" Nov 01 15:15:05.215780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.215757 5473 generic.go:356] "Generic (PLEG): container finished" podID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerID="7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5" exitCode=60 Nov 01 15:15:05.215883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.215840 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" event={"ID":"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b","Type":"ContainerDied","Data":"7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5"} Nov 01 15:15:05.216056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.216041 5473 scope.go:122] "RemoveContainer" containerID="7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5" Nov 01 15:15:05.216367 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.216343 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r_odh-ai-gateway-infra(c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" Nov 01 15:15:05.217900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.217877 5473 generic.go:356] "Generic (PLEG): container finished" podID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerID="420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1" exitCode=60 Nov 01 15:15:05.218003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.217959 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" event={"ID":"bb9ab947-ed5d-4f22-baf4-555f4da6fe44","Type":"ContainerDied","Data":"420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1"} Nov 01 15:15:05.218177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.218143 5473 scope.go:122] "RemoveContainer" containerID="420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1" Nov 01 15:15:05.218405 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.218368 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92_odh-ai-gateway-infra(bb9ab947-ed5d-4f22-baf4-555f4da6fe44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" Nov 01 15:15:05.220183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.220160 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerStarted","Data":"289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3"} Nov 01 15:15:05.222351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.222327 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerStarted","Data":"6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9"} Nov 01 15:15:05.224359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.224337 5473 generic.go:356] "Generic (PLEG): container finished" podID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerID="741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d" exitCode=60 Nov 01 15:15:05.224447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.224392 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" event={"ID":"24545ec6-f2fa-4658-9287-e915c228e8e0","Type":"ContainerDied","Data":"741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d"} Nov 01 15:15:05.224637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.224619 5473 scope.go:122] "RemoveContainer" containerID="741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d" Nov 01 15:15:05.224874 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:05.224850 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc_odh-ai-gateway-infra(24545ec6-f2fa-4658-9287-e915c228e8e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" Nov 01 15:15:05.228209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.228191 5473 scope.go:122] "RemoveContainer" containerID="aac4b2fb724b5fdee72963a998c46f8636bb43b4d0a9407b345e75b4eb66b5be" Nov 01 15:15:05.241559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.241540 5473 scope.go:122] "RemoveContainer" containerID="d80e510ee018fe2977908d7c08eeded29f1600343d7ab97682201541ebe3ac24" Nov 01 15:15:05.254941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.254919 5473 scope.go:122] "RemoveContainer" containerID="24a9f091b48e15392e33a83d52be77c4b63c96712d957957a84f355a7c6c32ea" Nov 01 15:15:05.268714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.268696 5473 scope.go:122] "RemoveContainer" containerID="872614f06a7a1acf27bcd464a90554161c28009bdd315ce980e6d47f5889af97" Nov 01 15:15:05.282129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.282106 5473 scope.go:122] "RemoveContainer" containerID="e64ce951590ebdaa5043ed66377691b75f6b567463c2f2a6edaa238ed70f1e34" Nov 01 15:15:05.295202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.295184 5473 scope.go:122] "RemoveContainer" containerID="7f70ee93dc919e93de18e0a400a4d10475c80a84a5461de54f239690e7e0ddc1" Nov 01 15:15:05.309001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.308973 5473 scope.go:122] "RemoveContainer" containerID="6ea70b3ce1d9ecb3ad8f382c816604ceac6cb249a8134ea3a24ac3a03fc5024d" Nov 01 15:15:05.323390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.323373 5473 scope.go:122] "RemoveContainer" containerID="ce96b9decc6e3664d983d0087d78bd58392edbae4588b2b27327cec14c099de4" Nov 01 15:15:05.341581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.341559 5473 scope.go:122] "RemoveContainer" containerID="78cb115ca19d7f1b9c62b437231ca459eb644a8b5cf66d779c4c6e919de15c99" Nov 01 15:15:05.381629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.381582 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" podStartSLOduration=5.381566726 podStartE2EDuration="5.381566726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:05.380332924 +0000 UTC m=+3444.438484775" watchObservedRunningTime="2026-11-01 15:15:05.381566726 +0000 UTC m=+3444.439718589" Nov 01 15:15:05.610557 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:05.610504 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" podStartSLOduration=5.610491559 podStartE2EDuration="5.610491559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:05.606788859 +0000 UTC m=+3444.664940719" watchObservedRunningTime="2026-11-01 15:15:05.610491559 +0000 UTC m=+3444.668643408" Nov 01 15:15:06.066460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.066408 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" podStartSLOduration=6.066393133 podStartE2EDuration="6.066393133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:06.063024039 +0000 UTC m=+3445.121175889" watchObservedRunningTime="2026-11-01 15:15:06.066393133 +0000 UTC m=+3445.124544997" Nov 01 15:15:06.235531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.235498 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerID="289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3" exitCode=60 Nov 01 15:15:06.235723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.235539 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerDied","Data":"289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3"} Nov 01 15:15:06.235723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.235603 5473 scope.go:122] "RemoveContainer" containerID="f71ef3941c09b0f1a19f544147e335d63879642d788935644e459c39b2f4093d" Nov 01 15:15:06.235877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.235861 5473 scope.go:122] "RemoveContainer" containerID="289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3" Nov 01 15:15:06.236274 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.236248 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp_odh-ai-gateway-infra(0ed0f6ce-0634-426b-a25a-b8dae42bb8d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" Nov 01 15:15:06.237640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.237611 5473 generic.go:356] "Generic (PLEG): container finished" podID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerID="6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9" exitCode=60 Nov 01 15:15:06.237728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.237657 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerDied","Data":"6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9"} Nov 01 15:15:06.237946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.237931 5473 scope.go:122] "RemoveContainer" containerID="6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9" Nov 01 15:15:06.238229 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.238199 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp_odh-ai-gateway-infra(8780c7c8-c128-4639-bcc0-7d806cbf5fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" Nov 01 15:15:06.239518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.239502 5473 scope.go:122] "RemoveContainer" containerID="741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d" Nov 01 15:15:06.239810 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.239791 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc_odh-ai-gateway-infra(24545ec6-f2fa-4658-9287-e915c228e8e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" Nov 01 15:15:06.241119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.241102 5473 scope.go:122] "RemoveContainer" containerID="93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617" Nov 01 15:15:06.241404 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.241382 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q_odh-ai-gateway-infra(0ebdb2c6-1b1f-4b97-b658-8f806387b52c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" Nov 01 15:15:06.242807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.242787 5473 generic.go:356] "Generic (PLEG): container finished" podID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerID="cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098" exitCode=60 Nov 01 15:15:06.242895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.242842 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerDied","Data":"cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098"} Nov 01 15:15:06.243080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.243064 5473 scope.go:122] "RemoveContainer" containerID="cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098" Nov 01 15:15:06.243319 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.243298 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd_odh-ai-gateway-infra(b417c672-9b62-45ba-9c03-1dc8eca1c809)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" Nov 01 15:15:06.244847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.244828 5473 scope.go:122] "RemoveContainer" containerID="4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06" Nov 01 15:15:06.245072 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.245048 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj_odh-ai-gateway-infra(874f4254-ae13-4ec0-8afb-c4342ca765ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" Nov 01 15:15:06.246480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.246460 5473 scope.go:122] "RemoveContainer" containerID="f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae" Nov 01 15:15:06.246695 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.246672 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv_odh-ai-gateway-infra(1538b70d-4c8e-4226-afc6-5a4bcaa6836d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" Nov 01 15:15:06.248313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.248293 5473 scope.go:122] "RemoveContainer" containerID="94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f" Nov 01 15:15:06.248547 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.248525 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58_odh-ai-gateway-infra(84b153d4-a17d-4cd2-ab3a-308be9b516f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" Nov 01 15:15:06.249983 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.249966 5473 scope.go:122] "RemoveContainer" containerID="c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818" Nov 01 15:15:06.250205 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.250185 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z_odh-ai-gateway-infra(731d4b63-f3ad-468a-a224-576f728e9e06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" Nov 01 15:15:06.250752 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.250736 5473 scope.go:122] "RemoveContainer" containerID="0b291947728b48c882238a60cea280fbaf23a48fa559db00a235c737c68757f5" Nov 01 15:15:06.251654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.251634 5473 scope.go:122] "RemoveContainer" containerID="4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f" Nov 01 15:15:06.252067 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.252046 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd_odh-ai-gateway-infra(02170a0e-d4fc-4024-b1a8-c5d8a4ce524f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" Nov 01 15:15:06.253371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.253326 5473 scope.go:122] "RemoveContainer" containerID="aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc" Nov 01 15:15:06.253540 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.253518 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9_odh-ai-gateway-infra(b2a7a920-701a-4c4f-bc06-bb73f00573e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" Nov 01 15:15:06.254911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.254893 5473 scope.go:122] "RemoveContainer" containerID="d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9" Nov 01 15:15:06.255108 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.255085 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws_odh-ai-gateway-infra(0cf4cfa9-eae6-43fc-9457-440c5940b693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" Nov 01 15:15:06.256513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.256493 5473 scope.go:122] "RemoveContainer" containerID="7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5" Nov 01 15:15:06.256724 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.256701 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r_odh-ai-gateway-infra(c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" Nov 01 15:15:06.258185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.258169 5473 scope.go:122] "RemoveContainer" containerID="420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1" Nov 01 15:15:06.258371 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:06.258354 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92_odh-ai-gateway-infra(bb9ab947-ed5d-4f22-baf4-555f4da6fe44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" Nov 01 15:15:06.264766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:06.264750 5473 scope.go:122] "RemoveContainer" containerID="2c30a5f044b9c9023bb32db9af97a34365bd58ff53bcb8cf8869a342f96d0c31" Nov 01 15:15:07.268461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:07.268435 5473 scope.go:122] "RemoveContainer" containerID="cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098" Nov 01 15:15:07.268881 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:07.268755 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd_odh-ai-gateway-infra(b417c672-9b62-45ba-9c03-1dc8eca1c809)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" Nov 01 15:15:07.270251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:07.270212 5473 scope.go:122] "RemoveContainer" containerID="289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3" Nov 01 15:15:07.270520 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:07.270500 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp_odh-ai-gateway-infra(0ed0f6ce-0634-426b-a25a-b8dae42bb8d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" Nov 01 15:15:07.271765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:07.271750 5473 scope.go:122] "RemoveContainer" containerID="6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9" Nov 01 15:15:07.271998 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:07.271981 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp_odh-ai-gateway-infra(8780c7c8-c128-4639-bcc0-7d806cbf5fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" Nov 01 15:15:07.622498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:07.622420 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:09.049961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:09.049927 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n"] Nov 01 15:15:09.054323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:09.054302 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a45376-4958n"] Nov 01 15:15:09.083515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:09.083487 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9843443-a8bf-4e3f-8cb5-da078eab35da" path="/var/lib/kubelet/pods/f9843443-a8bf-4e3f-8cb5-da078eab35da/volumes" Nov 01 15:15:12.225688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:12.225652 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:15.074933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:15.074894 5473 scope.go:122] "RemoveContainer" containerID="4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06" Nov 01 15:15:15.352888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:15.352848 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerStarted","Data":"9d8c5c8e75c2f61200a21fb191459b6481783809f2475be2b07b9c841d77a90d"} Nov 01 15:15:16.074637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.074606 5473 scope.go:122] "RemoveContainer" containerID="f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae" Nov 01 15:15:16.074885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.074867 5473 scope.go:122] "RemoveContainer" containerID="420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1" Nov 01 15:15:16.363490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.363422 5473 generic.go:356] "Generic (PLEG): container finished" podID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerID="9d8c5c8e75c2f61200a21fb191459b6481783809f2475be2b07b9c841d77a90d" exitCode=60 Nov 01 15:15:16.363851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.363489 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerDied","Data":"9d8c5c8e75c2f61200a21fb191459b6481783809f2475be2b07b9c841d77a90d"} Nov 01 15:15:16.363851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.363549 5473 scope.go:122] "RemoveContainer" containerID="4aeab46260d942f43e10758e4b8754e91687eef55d16fa37d1b916e101701a06" Nov 01 15:15:16.363958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.363918 5473 scope.go:122] "RemoveContainer" containerID="9d8c5c8e75c2f61200a21fb191459b6481783809f2475be2b07b9c841d77a90d" Nov 01 15:15:16.364260 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:16.364235 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj_odh-ai-gateway-infra(874f4254-ae13-4ec0-8afb-c4342ca765ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" Nov 01 15:15:16.479257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.479225 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj"] Nov 01 15:15:16.595132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:16.595099 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:17.075407 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.075379 5473 scope.go:122] "RemoveContainer" containerID="741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d" Nov 01 15:15:17.075542 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.075517 5473 scope.go:122] "RemoveContainer" containerID="aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc" Nov 01 15:15:17.075838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.075816 5473 scope.go:122] "RemoveContainer" containerID="289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3" Nov 01 15:15:17.076045 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.076025 5473 scope.go:122] "RemoveContainer" containerID="94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f" Nov 01 15:15:17.076328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.076265 5473 scope.go:122] "RemoveContainer" containerID="93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617" Nov 01 15:15:17.076738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.076393 5473 scope.go:122] "RemoveContainer" containerID="4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f" Nov 01 15:15:17.076738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.076506 5473 scope.go:122] "RemoveContainer" containerID="d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9" Nov 01 15:15:17.377357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.377329 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerStarted","Data":"387f2e0b61f780910e5a337783e554e52c7df5a5f2cecc56d2ae57bb25f5bfc2"} Nov 01 15:15:17.379982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.379955 5473 generic.go:356] "Generic (PLEG): container finished" podID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerID="7104782ede07942f9a9593882c72072c5c6c2e636d1c359b709b7a10f8dbef70" exitCode=60 Nov 01 15:15:17.380101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.380073 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" event={"ID":"bb9ab947-ed5d-4f22-baf4-555f4da6fe44","Type":"ContainerDied","Data":"7104782ede07942f9a9593882c72072c5c6c2e636d1c359b709b7a10f8dbef70"} Nov 01 15:15:17.380166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.380150 5473 scope.go:122] "RemoveContainer" containerID="420abeeecf2c83c3d2fdd0c5de076b109c6f550234de03689c0ac39fec53b4a1" Nov 01 15:15:17.380597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.380577 5473 scope.go:122] "RemoveContainer" containerID="7104782ede07942f9a9593882c72072c5c6c2e636d1c359b709b7a10f8dbef70" Nov 01 15:15:17.380943 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:17.380917 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92_odh-ai-gateway-infra(bb9ab947-ed5d-4f22-baf4-555f4da6fe44)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" Nov 01 15:15:17.661721 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.661697 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:17.765422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.765395 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca\") pod \"874f4254-ae13-4ec0-8afb-c4342ca765ed\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " Nov 01 15:15:17.765576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.765561 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk\" (UniqueName: \"kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk\") pod \"874f4254-ae13-4ec0-8afb-c4342ca765ed\" (UID: \"874f4254-ae13-4ec0-8afb-c4342ca765ed\") " Nov 01 15:15:17.765715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.765695 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca" pod "874f4254-ae13-4ec0-8afb-c4342ca765ed" (UID: "874f4254-ae13-4ec0-8afb-c4342ca765ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:17.766011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.765997 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874f4254-ae13-4ec0-8afb-c4342ca765ed-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:17.767380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.767361 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk" pod "874f4254-ae13-4ec0-8afb-c4342ca765ed" (UID: "874f4254-ae13-4ec0-8afb-c4342ca765ed"). InnerVolumeSpecName "kube-api-access-zlxmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:17.867623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:17.867593 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxmk\" (UniqueName: \"kubernetes.io/projected/874f4254-ae13-4ec0-8afb-c4342ca765ed-kube-api-access-zlxmk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:18.075012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.074977 5473 scope.go:122] "RemoveContainer" containerID="cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098" Nov 01 15:15:18.075196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.075091 5473 scope.go:122] "RemoveContainer" containerID="c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818" Nov 01 15:15:18.391516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.391488 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerID="c141ff50d2be78d46e0dceacf1975d17440adeb8cc7192048147c3124aa88d5b" exitCode=60 Nov 01 15:15:18.391930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.391562 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerDied","Data":"c141ff50d2be78d46e0dceacf1975d17440adeb8cc7192048147c3124aa88d5b"} Nov 01 15:15:18.391930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.391621 5473 scope.go:122] "RemoveContainer" containerID="289eb5394cff9942e74e2a4b8f9fc8eaf8dd7030f3d812257f4f672e805c5cb3" Nov 01 15:15:18.392077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.392061 5473 scope.go:122] "RemoveContainer" containerID="c141ff50d2be78d46e0dceacf1975d17440adeb8cc7192048147c3124aa88d5b" Nov 01 15:15:18.392423 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.392397 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp_odh-ai-gateway-infra(0ed0f6ce-0634-426b-a25a-b8dae42bb8d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" Nov 01 15:15:18.393770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.393746 5473 generic.go:356] "Generic (PLEG): container finished" podID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerID="3b1dfcb93e48d187c69b3cd9c08630efe62c026389d3e19d3c273f28b30be9ec" exitCode=60 Nov 01 15:15:18.393855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.393815 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" event={"ID":"24545ec6-f2fa-4658-9287-e915c228e8e0","Type":"ContainerDied","Data":"3b1dfcb93e48d187c69b3cd9c08630efe62c026389d3e19d3c273f28b30be9ec"} Nov 01 15:15:18.394253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.394234 5473 scope.go:122] "RemoveContainer" containerID="3b1dfcb93e48d187c69b3cd9c08630efe62c026389d3e19d3c273f28b30be9ec" Nov 01 15:15:18.394527 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.394510 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc_odh-ai-gateway-infra(24545ec6-f2fa-4658-9287-e915c228e8e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" Nov 01 15:15:18.395779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.395761 5473 generic.go:356] "Generic (PLEG): container finished" podID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerID="d63466fc3ae5925c5bd4b8e415a3e3220661985d52889fac0cdca464e192c646" exitCode=60 Nov 01 15:15:18.395850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.395830 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerDied","Data":"d63466fc3ae5925c5bd4b8e415a3e3220661985d52889fac0cdca464e192c646"} Nov 01 15:15:18.396249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.396225 5473 scope.go:122] "RemoveContainer" containerID="d63466fc3ae5925c5bd4b8e415a3e3220661985d52889fac0cdca464e192c646" Nov 01 15:15:18.396526 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.396503 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q_odh-ai-gateway-infra(0ebdb2c6-1b1f-4b97-b658-8f806387b52c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" Nov 01 15:15:18.397261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.397244 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" Nov 01 15:15:18.397353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.397285 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj" event={"ID":"874f4254-ae13-4ec0-8afb-c4342ca765ed","Type":"ContainerDied","Data":"f34194f263a5c8da7ea3d4a81c9b86ec16d8ab699505025c86edab9817b16770"} Nov 01 15:15:18.399107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.399090 5473 generic.go:356] "Generic (PLEG): container finished" podID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerID="387f2e0b61f780910e5a337783e554e52c7df5a5f2cecc56d2ae57bb25f5bfc2" exitCode=60 Nov 01 15:15:18.399202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.399157 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerDied","Data":"387f2e0b61f780910e5a337783e554e52c7df5a5f2cecc56d2ae57bb25f5bfc2"} Nov 01 15:15:18.399516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.399498 5473 scope.go:122] "RemoveContainer" containerID="387f2e0b61f780910e5a337783e554e52c7df5a5f2cecc56d2ae57bb25f5bfc2" Nov 01 15:15:18.399782 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.399758 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv_odh-ai-gateway-infra(1538b70d-4c8e-4226-afc6-5a4bcaa6836d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" Nov 01 15:15:18.401049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.401030 5473 generic.go:356] "Generic (PLEG): container finished" podID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerID="88366ed941f2594a6d36549c6920e7c9e7104e35008ce35fac852ccb1be555a8" exitCode=60 Nov 01 15:15:18.401137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.401097 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" event={"ID":"84b153d4-a17d-4cd2-ab3a-308be9b516f9","Type":"ContainerDied","Data":"88366ed941f2594a6d36549c6920e7c9e7104e35008ce35fac852ccb1be555a8"} Nov 01 15:15:18.401480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.401459 5473 scope.go:122] "RemoveContainer" containerID="88366ed941f2594a6d36549c6920e7c9e7104e35008ce35fac852ccb1be555a8" Nov 01 15:15:18.401699 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.401676 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58_odh-ai-gateway-infra(84b153d4-a17d-4cd2-ab3a-308be9b516f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" Nov 01 15:15:18.403062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.403042 5473 generic.go:356] "Generic (PLEG): container finished" podID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerID="4233ce585d45b2eedf0829bfa08ef2e0e525fdf5b42a0e45e0aec9a816ff700c" exitCode=60 Nov 01 15:15:18.403169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.403109 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" event={"ID":"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f","Type":"ContainerDied","Data":"4233ce585d45b2eedf0829bfa08ef2e0e525fdf5b42a0e45e0aec9a816ff700c"} Nov 01 15:15:18.403475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.403459 5473 scope.go:122] "RemoveContainer" containerID="4233ce585d45b2eedf0829bfa08ef2e0e525fdf5b42a0e45e0aec9a816ff700c" Nov 01 15:15:18.403683 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.403662 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd_odh-ai-gateway-infra(02170a0e-d4fc-4024-b1a8-c5d8a4ce524f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" Nov 01 15:15:18.404935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.404916 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerID="7b1d7e8a23c47c9a437485ca5dcf1db9b6821d0cab8e0bf21575ad4344afa327" exitCode=60 Nov 01 15:15:18.405018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.404984 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerDied","Data":"7b1d7e8a23c47c9a437485ca5dcf1db9b6821d0cab8e0bf21575ad4344afa327"} Nov 01 15:15:18.405301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.405283 5473 scope.go:122] "RemoveContainer" containerID="7b1d7e8a23c47c9a437485ca5dcf1db9b6821d0cab8e0bf21575ad4344afa327" Nov 01 15:15:18.405497 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.405480 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9_odh-ai-gateway-infra(b2a7a920-701a-4c4f-bc06-bb73f00573e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" Nov 01 15:15:18.407097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.407077 5473 generic.go:356] "Generic (PLEG): container finished" podID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerID="0894031c4ce4a788208bc0671ce55abac3825f9b79241328d0c3a9f7ee5624c9" exitCode=60 Nov 01 15:15:18.407166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.407137 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerDied","Data":"0894031c4ce4a788208bc0671ce55abac3825f9b79241328d0c3a9f7ee5624c9"} Nov 01 15:15:18.407461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.407436 5473 scope.go:122] "RemoveContainer" containerID="0894031c4ce4a788208bc0671ce55abac3825f9b79241328d0c3a9f7ee5624c9" Nov 01 15:15:18.407683 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:18.407658 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws_odh-ai-gateway-infra(0cf4cfa9-eae6-43fc-9457-440c5940b693)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" Nov 01 15:15:18.617009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.616979 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92"] Nov 01 15:15:18.701728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.701413 5473 scope.go:122] "RemoveContainer" containerID="741b04d5280530b27fbff68f235e7a8d9286edab5bf2160f5b88a89477d9c35d" Nov 01 15:15:18.721767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.721742 5473 scope.go:122] "RemoveContainer" containerID="93dbace5a1b1ac12b11b205b99c749c3a5e5a869d30898a8cabd54caa2245617" Nov 01 15:15:18.738652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.738625 5473 scope.go:122] "RemoveContainer" containerID="9d8c5c8e75c2f61200a21fb191459b6481783809f2475be2b07b9c841d77a90d" Nov 01 15:15:18.742735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.742709 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv"] Nov 01 15:15:18.754461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.754443 5473 scope.go:122] "RemoveContainer" containerID="f315f7a60ef1e7cdfa2efba64eaa935b99ea1d6abede713663b72a14c9fc65ae" Nov 01 15:15:18.787955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.787936 5473 scope.go:122] "RemoveContainer" containerID="94692d2c74414591efefe7d443fe98d46bfde28d68654e04f04141ddb2165c7f" Nov 01 15:15:18.790153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.790131 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:18.801230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.801196 5473 scope.go:122] "RemoveContainer" containerID="4d0b50fe7f51a464df49a5988e0ace2a8f6e1e710daa88ed8ab263f5ceb63e5f" Nov 01 15:15:18.814468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.814440 5473 scope.go:122] "RemoveContainer" containerID="aba298e92d814baa079e3fcf02ad41f72f8530784c32729553002caf67fa73cc" Nov 01 15:15:18.828462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.828442 5473 scope.go:122] "RemoveContainer" containerID="d2f31cc3f1d5246211ef87c0924cb5023626c2caa47597d4a69c8e379dcbc6c9" Nov 01 15:15:18.883158 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.883094 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca\") pod \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " Nov 01 15:15:18.883372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.883355 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w\" (UniqueName: \"kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w\") pod \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\" (UID: \"bb9ab947-ed5d-4f22-baf4-555f4da6fe44\") " Nov 01 15:15:18.883429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.883409 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca" pod "bb9ab947-ed5d-4f22-baf4-555f4da6fe44" (UID: "bb9ab947-ed5d-4f22-baf4-555f4da6fe44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:18.883793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.883775 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:18.885299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.885280 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w" pod "bb9ab947-ed5d-4f22-baf4-555f4da6fe44" (UID: "bb9ab947-ed5d-4f22-baf4-555f4da6fe44"). InnerVolumeSpecName "kube-api-access-dnn6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:18.985368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:18.985337 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnn6w\" (UniqueName: \"kubernetes.io/projected/bb9ab947-ed5d-4f22-baf4-555f4da6fe44-kube-api-access-dnn6w\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:19.074665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.074635 5473 scope.go:122] "RemoveContainer" containerID="7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5" Nov 01 15:15:19.423484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.423409 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerStarted","Data":"85dfecaa37364ac5f3066ddc2316ee73d06d1d4aebf110b0fe69e04990f9a7aa"} Nov 01 15:15:19.437331 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.437303 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerStarted","Data":"c7efb0b15730c86e9e1d4ae66ca20a7dad060174f890de1d6da569ac0c4f1094"} Nov 01 15:15:19.437830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.437707 5473 scope.go:122] "RemoveContainer" containerID="c7efb0b15730c86e9e1d4ae66ca20a7dad060174f890de1d6da569ac0c4f1094" Nov 01 15:15:19.438087 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:19.438065 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z_odh-ai-gateway-infra(731d4b63-f3ad-468a-a224-576f728e9e06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" Nov 01 15:15:19.445247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.445206 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" event={"ID":"bb9ab947-ed5d-4f22-baf4-555f4da6fe44","Type":"ContainerDied","Data":"100e31594ea95af04d84b8b9139194ff67616f4bfd421f094d89ac2ac0a4607e"} Nov 01 15:15:19.445384 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.445259 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92" Nov 01 15:15:19.445384 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.445281 5473 scope.go:122] "RemoveContainer" containerID="7104782ede07942f9a9593882c72072c5c6c2e636d1c359b709b7a10f8dbef70" Nov 01 15:15:19.491189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.491152 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj"] Nov 01 15:15:19.503425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.500447 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892435-8kddj"] Nov 01 15:15:19.528799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.528761 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp"] Nov 01 15:15:19.621286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.621260 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:19.685975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.685944 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:19.696227 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.696193 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf\" (UniqueName: \"kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf\") pod \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " Nov 01 15:15:19.696337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.696323 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca\") pod \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\" (UID: \"1538b70d-4c8e-4226-afc6-5a4bcaa6836d\") " Nov 01 15:15:19.696661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.696635 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca" pod "1538b70d-4c8e-4226-afc6-5a4bcaa6836d" (UID: "1538b70d-4c8e-4226-afc6-5a4bcaa6836d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:19.698098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.698078 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf" pod "1538b70d-4c8e-4226-afc6-5a4bcaa6836d" (UID: "1538b70d-4c8e-4226-afc6-5a4bcaa6836d"). InnerVolumeSpecName "kube-api-access-twvqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:19.761478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.761452 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9"] Nov 01 15:15:19.797468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.797436 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca\") pod \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " Nov 01 15:15:19.797639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.797555 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874\" (UniqueName: \"kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874\") pod \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\" (UID: \"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8\") " Nov 01 15:15:19.797902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.797876 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca" pod "0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" (UID: "0ed0f6ce-0634-426b-a25a-b8dae42bb8d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:19.798251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.798231 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:19.798340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.798256 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:19.798340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.798270 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twvqf\" (UniqueName: \"kubernetes.io/projected/1538b70d-4c8e-4226-afc6-5a4bcaa6836d-kube-api-access-twvqf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:19.800146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.800121 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874" pod "0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" (UID: "0ed0f6ce-0634-426b-a25a-b8dae42bb8d8"). InnerVolumeSpecName "kube-api-access-mm874". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:19.912234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.900259 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm874\" (UniqueName: \"kubernetes.io/projected/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8-kube-api-access-mm874\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:19.927189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.919571 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd"] Nov 01 15:15:19.971554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.971506 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92"] Nov 01 15:15:19.978028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.977998 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8fd363-29892435-9hj92"] Nov 01 15:15:19.985080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.985047 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:19.998258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:19.996827 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws"] Nov 01 15:15:20.090347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.088617 5473 scope.go:122] "RemoveContainer" containerID="6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9" Nov 01 15:15:20.128237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.114624 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78"] Nov 01 15:15:20.128237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.114693 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ae9a22-xlj78"] Nov 01 15:15:20.128237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.115118 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx\" (UniqueName: \"kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx\") pod \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " Nov 01 15:15:20.128237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.115448 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca\") pod \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\" (UID: \"b2a7a920-701a-4c4f-bc06-bb73f00573e1\") " Nov 01 15:15:20.160310 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.160273 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc"] Nov 01 15:15:20.161323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.161289 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca" pod "b2a7a920-701a-4c4f-bc06-bb73f00573e1" (UID: "b2a7a920-701a-4c4f-bc06-bb73f00573e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:20.171525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.169683 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx" pod "b2a7a920-701a-4c4f-bc06-bb73f00573e1" (UID: "b2a7a920-701a-4c4f-bc06-bb73f00573e1"). InnerVolumeSpecName "kube-api-access-s42vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:20.241710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.239833 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42vx\" (UniqueName: \"kubernetes.io/projected/b2a7a920-701a-4c4f-bc06-bb73f00573e1-kube-api-access-s42vx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.241710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.239871 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2a7a920-701a-4c4f-bc06-bb73f00573e1-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.250648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.249254 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58"] Nov 01 15:15:20.303967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.303935 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:20.344344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.343635 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q"] Nov 01 15:15:20.392520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.392493 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:20.446111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.446074 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca" pod "02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" (UID: "02170a0e-d4fc-4024-b1a8-c5d8a4ce524f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:20.463480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.463451 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca\") pod \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " Nov 01 15:15:20.463840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.463812 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt\" (UniqueName: \"kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt\") pod \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\" (UID: \"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f\") " Nov 01 15:15:20.464796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.464770 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.466279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.466032 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" event={"ID":"0ed0f6ce-0634-426b-a25a-b8dae42bb8d8","Type":"ContainerDied","Data":"b4c8ede60acb58ae9493a0b408f398fe654754ce62ff13fa1d7c45633c081ab8"} Nov 01 15:15:20.466279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.466101 5473 scope.go:122] "RemoveContainer" containerID="c141ff50d2be78d46e0dceacf1975d17440adeb8cc7192048147c3124aa88d5b" Nov 01 15:15:20.466279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.466203 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp" Nov 01 15:15:20.476779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.467805 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt" pod "02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" (UID: "02170a0e-d4fc-4024-b1a8-c5d8a4ce524f"). InnerVolumeSpecName "kube-api-access-mjwkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:20.498563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.498527 5473 generic.go:356] "Generic (PLEG): container finished" podID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerID="85dfecaa37364ac5f3066ddc2316ee73d06d1d4aebf110b0fe69e04990f9a7aa" exitCode=60 Nov 01 15:15:20.508119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.498706 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerDied","Data":"85dfecaa37364ac5f3066ddc2316ee73d06d1d4aebf110b0fe69e04990f9a7aa"} Nov 01 15:15:20.508119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.505818 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:20.508119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.507282 5473 scope.go:122] "RemoveContainer" containerID="85dfecaa37364ac5f3066ddc2316ee73d06d1d4aebf110b0fe69e04990f9a7aa" Nov 01 15:15:20.508119 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:20.507692 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd_odh-ai-gateway-infra(b417c672-9b62-45ba-9c03-1dc8eca1c809)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" Nov 01 15:15:20.516671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.515131 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" event={"ID":"1538b70d-4c8e-4226-afc6-5a4bcaa6836d","Type":"ContainerDied","Data":"127b4a49a60d4f49a57d2523d74967d3f0f1f635135fa435e01dd5439c084771"} Nov 01 15:15:20.516671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.515265 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv" Nov 01 15:15:20.523368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.523202 5473 scope.go:122] "RemoveContainer" containerID="cad5408921d32756bc4398d7c0893cbe49f76cf9901c19e11c415d658155d098" Nov 01 15:15:20.523829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.523810 5473 generic.go:356] "Generic (PLEG): container finished" podID="731d4b63-f3ad-468a-a224-576f728e9e06" containerID="c7efb0b15730c86e9e1d4ae66ca20a7dad060174f890de1d6da569ac0c4f1094" exitCode=60 Nov 01 15:15:20.524230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.523900 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerDied","Data":"c7efb0b15730c86e9e1d4ae66ca20a7dad060174f890de1d6da569ac0c4f1094"} Nov 01 15:15:20.526315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.526300 5473 scope.go:122] "RemoveContainer" containerID="c7efb0b15730c86e9e1d4ae66ca20a7dad060174f890de1d6da569ac0c4f1094" Nov 01 15:15:20.527002 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:20.526982 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z_odh-ai-gateway-infra(731d4b63-f3ad-468a-a224-576f728e9e06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" Nov 01 15:15:20.527460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.527444 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" Nov 01 15:15:20.528124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.527937 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:20.528124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.527590 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd" event={"ID":"02170a0e-d4fc-4024-b1a8-c5d8a4ce524f","Type":"ContainerDied","Data":"a243caa5cc9dedc810c6223cd7cecede0b22ebac7ef8388d92f08544081652fd"} Nov 01 15:15:20.536299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.529750 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" event={"ID":"b2a7a920-701a-4c4f-bc06-bb73f00573e1","Type":"ContainerDied","Data":"b72216238bd3dff2e8b9d2000f082a72b4e5d9fbe8c34f674c9bc25c45404541"} Nov 01 15:15:20.536484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.536463 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" event={"ID":"0cf4cfa9-eae6-43fc-9457-440c5940b693","Type":"ContainerDied","Data":"9d1806c82230d91c4dafca10e79f9b628dabc967662f42d33ef3fc2a400c0742"} Nov 01 15:15:20.540228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.537108 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws" Nov 01 15:15:20.540228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.537307 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9" Nov 01 15:15:20.544829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.543984 5473 scope.go:122] "RemoveContainer" containerID="387f2e0b61f780910e5a337783e554e52c7df5a5f2cecc56d2ae57bb25f5bfc2" Nov 01 15:15:20.546776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.546248 5473 generic.go:356] "Generic (PLEG): container finished" podID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerID="de473d80c508cd8a213690d8bc7186f0d36723b6c57a423821e9dbd732702b81" exitCode=60 Nov 01 15:15:20.546776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.546354 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" event={"ID":"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b","Type":"ContainerDied","Data":"de473d80c508cd8a213690d8bc7186f0d36723b6c57a423821e9dbd732702b81"} Nov 01 15:15:20.546776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.546772 5473 scope.go:122] "RemoveContainer" containerID="de473d80c508cd8a213690d8bc7186f0d36723b6c57a423821e9dbd732702b81" Nov 01 15:15:20.547180 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:20.547121 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r_odh-ai-gateway-infra(c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" Nov 01 15:15:20.560091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.560073 5473 scope.go:122] "RemoveContainer" containerID="c49d0e62508e2c826e52646cffdc3e316db3f814d6219cb31ed2ae9bada4c818" Nov 01 15:15:20.565953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.565931 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472\" (UniqueName: \"kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472\") pod \"0cf4cfa9-eae6-43fc-9457-440c5940b693\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " Nov 01 15:15:20.566284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.566255 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca\") pod \"0cf4cfa9-eae6-43fc-9457-440c5940b693\" (UID: \"0cf4cfa9-eae6-43fc-9457-440c5940b693\") " Nov 01 15:15:20.566623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.566599 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca" pod "0cf4cfa9-eae6-43fc-9457-440c5940b693" (UID: "0cf4cfa9-eae6-43fc-9457-440c5940b693"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:20.567167 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.567140 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf4cfa9-eae6-43fc-9457-440c5940b693-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.567359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.567208 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwkt\" (UniqueName: \"kubernetes.io/projected/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f-kube-api-access-mjwkt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.569899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.569874 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472" pod "0cf4cfa9-eae6-43fc-9457-440c5940b693" (UID: "0cf4cfa9-eae6-43fc-9457-440c5940b693"). InnerVolumeSpecName "kube-api-access-sf472". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:20.575993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.575971 5473 scope.go:122] "RemoveContainer" containerID="4233ce585d45b2eedf0829bfa08ef2e0e525fdf5b42a0e45e0aec9a816ff700c" Nov 01 15:15:20.586996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.586973 5473 scope.go:122] "RemoveContainer" containerID="7b1d7e8a23c47c9a437485ca5dcf1db9b6821d0cab8e0bf21575ad4344afa327" Nov 01 15:15:20.602694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.602673 5473 scope.go:122] "RemoveContainer" containerID="0894031c4ce4a788208bc0671ce55abac3825f9b79241328d0c3a9f7ee5624c9" Nov 01 15:15:20.602783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.602684 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:20.613152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.613134 5473 scope.go:122] "RemoveContainer" containerID="7bd61f93189d04a9f3d1d01e581a9a419ae461b23bb120af05874315b6bcfed5" Nov 01 15:15:20.668081 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668054 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4\" (UniqueName: \"kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4\") pod \"24545ec6-f2fa-4658-9287-e915c228e8e0\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " Nov 01 15:15:20.668190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668099 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca\") pod \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " Nov 01 15:15:20.668190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668123 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca\") pod \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " Nov 01 15:15:20.668190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668146 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77\" (UniqueName: \"kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77\") pod \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\" (UID: \"0ebdb2c6-1b1f-4b97-b658-8f806387b52c\") " Nov 01 15:15:20.668387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668202 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca\") pod \"24545ec6-f2fa-4658-9287-e915c228e8e0\" (UID: \"24545ec6-f2fa-4658-9287-e915c228e8e0\") " Nov 01 15:15:20.668387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668366 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v\" (UniqueName: \"kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v\") pod \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\" (UID: \"84b153d4-a17d-4cd2-ab3a-308be9b516f9\") " Nov 01 15:15:20.668518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668490 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca" pod "84b153d4-a17d-4cd2-ab3a-308be9b516f9" (UID: "84b153d4-a17d-4cd2-ab3a-308be9b516f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:20.668612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668591 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca" pod "0ebdb2c6-1b1f-4b97-b658-8f806387b52c" (UID: "0ebdb2c6-1b1f-4b97-b658-8f806387b52c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:20.668708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668609 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca" pod "24545ec6-f2fa-4658-9287-e915c228e8e0" (UID: "24545ec6-f2fa-4658-9287-e915c228e8e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:20.668951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668930 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24545ec6-f2fa-4658-9287-e915c228e8e0-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.669047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668956 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf472\" (UniqueName: \"kubernetes.io/projected/0cf4cfa9-eae6-43fc-9457-440c5940b693-kube-api-access-sf472\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.669047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668972 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84b153d4-a17d-4cd2-ab3a-308be9b516f9-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.669047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.668985 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.670175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.670155 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4" pod "24545ec6-f2fa-4658-9287-e915c228e8e0" (UID: "24545ec6-f2fa-4658-9287-e915c228e8e0"). InnerVolumeSpecName "kube-api-access-vf5z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:20.670863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.670841 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v" pod "84b153d4-a17d-4cd2-ab3a-308be9b516f9" (UID: "84b153d4-a17d-4cd2-ab3a-308be9b516f9"). InnerVolumeSpecName "kube-api-access-m8j4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:20.670926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.670878 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77" pod "0ebdb2c6-1b1f-4b97-b658-8f806387b52c" (UID: "0ebdb2c6-1b1f-4b97-b658-8f806387b52c"). InnerVolumeSpecName "kube-api-access-mxt77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:20.738613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.738534 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z"] Nov 01 15:15:20.770164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.770133 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5z4\" (UniqueName: \"kubernetes.io/projected/24545ec6-f2fa-4658-9287-e915c228e8e0-kube-api-access-vf5z4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.770164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.770159 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxt77\" (UniqueName: \"kubernetes.io/projected/0ebdb2c6-1b1f-4b97-b658-8f806387b52c-kube-api-access-mxt77\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:20.770341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:20.770178 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8j4v\" (UniqueName: \"kubernetes.io/projected/84b153d4-a17d-4cd2-ab3a-308be9b516f9-kube-api-access-m8j4v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:21.061364 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.061040 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:21.107949 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.107904 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef18acc-c96e-4579-9652-794c0daec4b9" path="/var/lib/kubelet/pods/1ef18acc-c96e-4579-9652-794c0daec4b9/volumes" Nov 01 15:15:21.108749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.108726 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" path="/var/lib/kubelet/pods/874f4254-ae13-4ec0-8afb-c4342ca765ed/volumes" Nov 01 15:15:21.109956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.109931 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" path="/var/lib/kubelet/pods/bb9ab947-ed5d-4f22-baf4-555f4da6fe44/volumes" Nov 01 15:15:21.178605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.178575 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp"] Nov 01 15:15:21.184825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.184796 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-536a87-29892435-f4gpp"] Nov 01 15:15:21.256367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.256339 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd"] Nov 01 15:15:21.289698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.289639 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv"] Nov 01 15:15:21.293667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.293642 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-15ad1e-29892435-p5bjv"] Nov 01 15:15:21.562326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.562296 5473 generic.go:356] "Generic (PLEG): container finished" podID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerID="fd3709765b7920983d084fe2b23df852124b209532549a55430097bfe2399e47" exitCode=60 Nov 01 15:15:21.562799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.562364 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerDied","Data":"fd3709765b7920983d084fe2b23df852124b209532549a55430097bfe2399e47"} Nov 01 15:15:21.562799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.562419 5473 scope.go:122] "RemoveContainer" containerID="6321c60b9f3358cfcf9943828c8c56133c3278afcc7059e88dba7cbcfb5af6b9" Nov 01 15:15:21.562908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.562848 5473 scope.go:122] "RemoveContainer" containerID="fd3709765b7920983d084fe2b23df852124b209532549a55430097bfe2399e47" Nov 01 15:15:21.563251 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:21.563192 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp_odh-ai-gateway-infra(8780c7c8-c128-4639-bcc0-7d806cbf5fb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" Nov 01 15:15:21.563966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.563948 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" Nov 01 15:15:21.564088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.563973 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc" event={"ID":"24545ec6-f2fa-4658-9287-e915c228e8e0","Type":"ContainerDied","Data":"2af950df38a31a720464bd30077d3954647f706a156a52c3dbfa1d399a76779f"} Nov 01 15:15:21.565580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.565559 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" event={"ID":"0ebdb2c6-1b1f-4b97-b658-8f806387b52c","Type":"ContainerDied","Data":"51515ec7522fee927b0acac6a07615f38f8fa8ed9068a6719d18306246df280e"} Nov 01 15:15:21.565691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.565584 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q" Nov 01 15:15:21.570335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.570313 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" Nov 01 15:15:21.570422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.570357 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58" event={"ID":"84b153d4-a17d-4cd2-ab3a-308be9b516f9","Type":"ContainerDied","Data":"a18d3c02f31c172f53582b91585a41037be0493f1bf4127dfd57308463a8be99"} Nov 01 15:15:21.579183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.579160 5473 scope.go:122] "RemoveContainer" containerID="3b1dfcb93e48d187c69b3cd9c08630efe62c026389d3e19d3c273f28b30be9ec" Nov 01 15:15:21.591073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.591050 5473 scope.go:122] "RemoveContainer" containerID="d63466fc3ae5925c5bd4b8e415a3e3220661985d52889fac0cdca464e192c646" Nov 01 15:15:21.609977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.609954 5473 scope.go:122] "RemoveContainer" containerID="88366ed941f2594a6d36549c6920e7c9e7104e35008ce35fac852ccb1be555a8" Nov 01 15:15:21.631396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.631372 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9"] Nov 01 15:15:21.636758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.636681 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c89271-29892435-ddsc9"] Nov 01 15:15:21.757013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.756986 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:21.797359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.797330 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:21.801518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.801496 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5\" (UniqueName: \"kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5\") pod \"731d4b63-f3ad-468a-a224-576f728e9e06\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " Nov 01 15:15:21.801639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.801587 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca\") pod \"731d4b63-f3ad-468a-a224-576f728e9e06\" (UID: \"731d4b63-f3ad-468a-a224-576f728e9e06\") " Nov 01 15:15:21.801953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.801916 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca" pod "731d4b63-f3ad-468a-a224-576f728e9e06" (UID: "731d4b63-f3ad-468a-a224-576f728e9e06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:21.802114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.802098 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731d4b63-f3ad-468a-a224-576f728e9e06-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:21.803847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.803820 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5" pod "731d4b63-f3ad-468a-a224-576f728e9e06" (UID: "731d4b63-f3ad-468a-a224-576f728e9e06"). InnerVolumeSpecName "kube-api-access-v6gk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:21.807628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.807597 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd"] Nov 01 15:15:21.811151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.811130 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5b0dcc-29892435-fz5fd"] Nov 01 15:15:21.902570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.902505 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca\") pod \"b417c672-9b62-45ba-9c03-1dc8eca1c809\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " Nov 01 15:15:21.902722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.902704 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm\" (UniqueName: \"kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm\") pod \"b417c672-9b62-45ba-9c03-1dc8eca1c809\" (UID: \"b417c672-9b62-45ba-9c03-1dc8eca1c809\") " Nov 01 15:15:21.902933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.902909 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca" pod "b417c672-9b62-45ba-9c03-1dc8eca1c809" (UID: "b417c672-9b62-45ba-9c03-1dc8eca1c809"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:21.903163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.903146 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b417c672-9b62-45ba-9c03-1dc8eca1c809-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:21.903236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.903165 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gk5\" (UniqueName: \"kubernetes.io/projected/731d4b63-f3ad-468a-a224-576f728e9e06-kube-api-access-v6gk5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:21.904854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.904830 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm" pod "b417c672-9b62-45ba-9c03-1dc8eca1c809" (UID: "b417c672-9b62-45ba-9c03-1dc8eca1c809"). InnerVolumeSpecName "kube-api-access-6vhxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:21.981654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.981622 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws"] Nov 01 15:15:21.985251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:21.985226 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892435-dwzws"] Nov 01 15:15:22.004635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.004609 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhxm\" (UniqueName: \"kubernetes.io/projected/b417c672-9b62-45ba-9c03-1dc8eca1c809-kube-api-access-6vhxm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:22.526632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.526599 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r"] Nov 01 15:15:22.588819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.588785 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" event={"ID":"731d4b63-f3ad-468a-a224-576f728e9e06","Type":"ContainerDied","Data":"0eb477dd6461fbb41f72ba800bc6e63991a932d57548747691c41dfe470dbf11"} Nov 01 15:15:22.589295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.588849 5473 scope.go:122] "RemoveContainer" containerID="c7efb0b15730c86e9e1d4ae66ca20a7dad060174f890de1d6da569ac0c4f1094" Nov 01 15:15:22.589295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.588940 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z" Nov 01 15:15:22.604543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.604514 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" event={"ID":"b417c672-9b62-45ba-9c03-1dc8eca1c809","Type":"ContainerDied","Data":"ecc99acc7a007c52b4a3b09854147acd224cc22b3e837f12606b1f8170b186b1"} Nov 01 15:15:22.604647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.604628 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd" Nov 01 15:15:22.609130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.608408 5473 scope.go:122] "RemoveContainer" containerID="85dfecaa37364ac5f3066ddc2316ee73d06d1d4aebf110b0fe69e04990f9a7aa" Nov 01 15:15:22.725152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.725123 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:22.788727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.788659 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc"] Nov 01 15:15:22.792652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.792629 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892435-ssxmc"] Nov 01 15:15:22.816298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.816269 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca\") pod \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " Nov 01 15:15:22.816475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.816459 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd\" (UniqueName: \"kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd\") pod \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\" (UID: \"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b\") " Nov 01 15:15:22.816594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.816573 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca" pod "c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" (UID: "c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:22.816938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.816919 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:22.818455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.818434 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd" pod "c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" (UID: "c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b"). InnerVolumeSpecName "kube-api-access-wz8dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:22.918353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:22.918326 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz8dd\" (UniqueName: \"kubernetes.io/projected/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b-kube-api-access-wz8dd\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:23.016138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.016107 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58"] Nov 01 15:15:23.025168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.025141 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892435-rvt58"] Nov 01 15:15:23.086297 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.086034 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" path="/var/lib/kubelet/pods/02170a0e-d4fc-4024-b1a8-c5d8a4ce524f/volumes" Nov 01 15:15:23.087325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.087300 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" path="/var/lib/kubelet/pods/0cf4cfa9-eae6-43fc-9457-440c5940b693/volumes" Nov 01 15:15:23.088229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.088195 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" path="/var/lib/kubelet/pods/0ed0f6ce-0634-426b-a25a-b8dae42bb8d8/volumes" Nov 01 15:15:23.089415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.089396 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" path="/var/lib/kubelet/pods/1538b70d-4c8e-4226-afc6-5a4bcaa6836d/volumes" Nov 01 15:15:23.090182 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.090163 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" path="/var/lib/kubelet/pods/24545ec6-f2fa-4658-9287-e915c228e8e0/volumes" Nov 01 15:15:23.090975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.090957 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" path="/var/lib/kubelet/pods/84b153d4-a17d-4cd2-ab3a-308be9b516f9/volumes" Nov 01 15:15:23.092135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.092116 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" path="/var/lib/kubelet/pods/b2a7a920-701a-4c4f-bc06-bb73f00573e1/volumes" Nov 01 15:15:23.141455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.141425 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q"] Nov 01 15:15:23.145836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.145812 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892435-x2p7q"] Nov 01 15:15:23.260806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.260778 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z"] Nov 01 15:15:23.264041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.263998 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892435-xrm9z"] Nov 01 15:15:23.370730 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.370656 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd"] Nov 01 15:15:23.374056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.374033 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2cf1ee-29892435-jjcbd"] Nov 01 15:15:23.630717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.630642 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" Nov 01 15:15:23.630717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.630654 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r" event={"ID":"c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b","Type":"ContainerDied","Data":"814ba5b4bd5e6d9274f45ee7c50c042251a11109834f216153db75667f54b716"} Nov 01 15:15:23.631169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.630719 5473 scope.go:122] "RemoveContainer" containerID="de473d80c508cd8a213690d8bc7186f0d36723b6c57a423821e9dbd732702b81" Nov 01 15:15:23.797614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.797577 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r"] Nov 01 15:15:23.800923 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.800901 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ea1620-29892435-lgx2r"] Nov 01 15:15:23.914513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:23.914430 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp"] Nov 01 15:15:24.070970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.070942 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:24.133656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.133628 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca\") pod \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " Nov 01 15:15:24.133832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.133814 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl\" (UniqueName: \"kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl\") pod \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\" (UID: \"8780c7c8-c128-4639-bcc0-7d806cbf5fb9\") " Nov 01 15:15:24.134022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.133998 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca" pod "8780c7c8-c128-4639-bcc0-7d806cbf5fb9" (UID: "8780c7c8-c128-4639-bcc0-7d806cbf5fb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:24.134408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.134380 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:24.135823 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.135803 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl" pod "8780c7c8-c128-4639-bcc0-7d806cbf5fb9" (UID: "8780c7c8-c128-4639-bcc0-7d806cbf5fb9"). InnerVolumeSpecName "kube-api-access-rzdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:24.235703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.235628 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzdcl\" (UniqueName: \"kubernetes.io/projected/8780c7c8-c128-4639-bcc0-7d806cbf5fb9-kube-api-access-rzdcl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:24.639530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.639502 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" Nov 01 15:15:24.640026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.639536 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp" event={"ID":"8780c7c8-c128-4639-bcc0-7d806cbf5fb9","Type":"ContainerDied","Data":"ea204c93d4a48c37683b48567ca833b5dd3990c20c9d8ecf0f2c6deb69016a6e"} Nov 01 15:15:24.640026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.639598 5473 scope.go:122] "RemoveContainer" containerID="fd3709765b7920983d084fe2b23df852124b209532549a55430097bfe2399e47" Nov 01 15:15:24.806860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.806829 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp"] Nov 01 15:15:24.810127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:24.810101 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ca0f9e-29892435-ntzjp"] Nov 01 15:15:25.083307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.083269 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" path="/var/lib/kubelet/pods/0ebdb2c6-1b1f-4b97-b658-8f806387b52c/volumes" Nov 01 15:15:25.083829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.083807 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" path="/var/lib/kubelet/pods/731d4b63-f3ad-468a-a224-576f728e9e06/volumes" Nov 01 15:15:25.084318 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.084299 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" path="/var/lib/kubelet/pods/8780c7c8-c128-4639-bcc0-7d806cbf5fb9/volumes" Nov 01 15:15:25.085041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.085020 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" path="/var/lib/kubelet/pods/b417c672-9b62-45ba-9c03-1dc8eca1c809/volumes" Nov 01 15:15:25.085510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.085491 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" path="/var/lib/kubelet/pods/c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b/volumes" Nov 01 15:15:25.634458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.634425 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:25.720701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.720671 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw"] Nov 01 15:15:25.721509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721488 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:25.721509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721509 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721530 5473 cpu_manager.go:414] "containerMap: removing container" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721538 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721551 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721558 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721569 5473 cpu_manager.go:414] "containerMap: removing container" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721576 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721585 5473 cpu_manager.go:414] "containerMap: removing container" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721592 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721605 5473 cpu_manager.go:414] "containerMap: removing container" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721612 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721621 5473 cpu_manager.go:414] "containerMap: removing container" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721627 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721645 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721652 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721662 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721669 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721691 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.721700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721698 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721714 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721720 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721731 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721738 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721754 5473 cpu_manager.go:414] "containerMap: removing container" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721760 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721769 5473 cpu_manager.go:414] "containerMap: removing container" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721775 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721789 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721796 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721806 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721813 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721823 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721829 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721841 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721848 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721863 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721870 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721889 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721896 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721906 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721913 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721922 5473 cpu_manager.go:414] "containerMap: removing container" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721929 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721938 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721945 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721957 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721964 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721975 5473 cpu_manager.go:414] "containerMap: removing container" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721982 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.721993 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722001 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722016 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722024 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722035 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722042 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722053 5473 cpu_manager.go:414] "containerMap: removing container" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722060 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722073 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722080 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722099 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722107 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722117 5473 cpu_manager.go:414] "containerMap: removing container" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722124 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722136 5473 cpu_manager.go:414] "containerMap: removing container" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722143 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722156 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722163 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722333 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722348 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722360 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722370 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722381 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722393 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722402 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722413 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722423 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722431 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722443 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722453 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722464 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b417c672-9b62-45ba-9c03-1dc8eca1c809" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722473 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722483 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722492 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722500 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722509 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722517 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722527 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722535 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722543 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8780c7c8-c128-4639-bcc0-7d806cbf5fb9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722556 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722565 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed0f6ce-0634-426b-a25a-b8dae42bb8d8" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722575 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722583 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722593 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722604 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722613 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722623 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb9ab947-ed5d-4f22-baf4-555f4da6fe44" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722632 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722640 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722649 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722660 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722668 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:25.722682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.722676 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:25.728512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.728492 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.778152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.778125 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw"] Nov 01 15:15:25.785668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.785646 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:15:25.785806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.785708 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:15:25.857123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.857094 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.857302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.857169 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.857352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.857319 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjp8p\" (UniqueName: \"kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.958562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.958488 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.958722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.958567 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.958722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.958607 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp8p\" (UniqueName: \"kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.959112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.959090 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:25.960958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:25.960937 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:26.018143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:26.018111 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp8p\" (UniqueName: \"kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p\") pod \"maas-api-revoke-sub-d5bd84c1-wp9pw\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:26.042806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:26.042779 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:26.415092 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:26.415053 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7644595_12fe_48ac_92c8_765cc57d4a4c.slice/crio-16b0f4bb174072f77518a543d533719364fd6a819076c07e027d27b2edc920b2 WatchSource:0}: Error finding container 16b0f4bb174072f77518a543d533719364fd6a819076c07e027d27b2edc920b2: Status 404 returned error can't find the container with id 16b0f4bb174072f77518a543d533719364fd6a819076c07e027d27b2edc920b2 Nov 01 15:15:26.421867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:26.421830 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw"] Nov 01 15:15:26.672300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:26.672199 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" event={"ID":"a7644595-12fe-48ac-92c8-765cc57d4a4c","Type":"ContainerStarted","Data":"16b0f4bb174072f77518a543d533719364fd6a819076c07e027d27b2edc920b2"} Nov 01 15:15:27.682287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:27.682194 5473 generic.go:356] "Generic (PLEG): container finished" podID="a7644595-12fe-48ac-92c8-765cc57d4a4c" containerID="8a178274fae497b98af92a5e15d74595c90e7d3c612d4e3ad4fa831845c83cf8" exitCode=0 Nov 01 15:15:27.682287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:27.682251 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" event={"ID":"a7644595-12fe-48ac-92c8-765cc57d4a4c","Type":"ContainerDied","Data":"8a178274fae497b98af92a5e15d74595c90e7d3c612d4e3ad4fa831845c83cf8"} Nov 01 15:15:28.847754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.847731 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:28.995577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.995488 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p\" (UniqueName: \"kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p\") pod \"a7644595-12fe-48ac-92c8-765cc57d4a4c\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " Nov 01 15:15:28.995577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.995556 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca\") pod \"a7644595-12fe-48ac-92c8-765cc57d4a4c\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " Nov 01 15:15:28.995877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.995607 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca\") pod \"a7644595-12fe-48ac-92c8-765cc57d4a4c\" (UID: \"a7644595-12fe-48ac-92c8-765cc57d4a4c\") " Nov 01 15:15:28.996055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.996002 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca" pod "a7644595-12fe-48ac-92c8-765cc57d4a4c" (UID: "a7644595-12fe-48ac-92c8-765cc57d4a4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:28.996391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.996364 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:28.998090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.998060 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p" pod "a7644595-12fe-48ac-92c8-765cc57d4a4c" (UID: "a7644595-12fe-48ac-92c8-765cc57d4a4c"). InnerVolumeSpecName "kube-api-access-zjp8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:28.998648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:28.998623 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca" pod "a7644595-12fe-48ac-92c8-765cc57d4a4c" (UID: "a7644595-12fe-48ac-92c8-765cc57d4a4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:15:29.097355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:29.097331 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjp8p\" (UniqueName: \"kubernetes.io/projected/a7644595-12fe-48ac-92c8-765cc57d4a4c-kube-api-access-zjp8p\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:29.097464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:29.097357 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7644595-12fe-48ac-92c8-765cc57d4a4c-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:29.700033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:29.699996 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" Nov 01 15:15:29.700187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:29.700042 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw" event={"ID":"a7644595-12fe-48ac-92c8-765cc57d4a4c","Type":"ContainerDied","Data":"16b0f4bb174072f77518a543d533719364fd6a819076c07e027d27b2edc920b2"} Nov 01 15:15:29.700187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:29.700092 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16b0f4bb174072f77518a543d533719364fd6a819076c07e027d27b2edc920b2" Nov 01 15:15:30.103745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:30.103708 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:31.111393 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:31.111362 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:15:34.531844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:34.531809 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:36.416497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.416467 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt"] Nov 01 15:15:36.417111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417098 5473 cpu_manager.go:414] "containerMap: removing container" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:36.417157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417113 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:36.417157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417126 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:36.417157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417131 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:36.417157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417147 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:36.417157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417152 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf4cfa9-eae6-43fc-9457-440c5940b693" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417168 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417173 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417192 5473 cpu_manager.go:414] "containerMap: removing container" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417197 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02170a0e-d4fc-4024-b1a8-c5d8a4ce524f" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417206 5473 cpu_manager.go:414] "containerMap: removing container" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417210 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731d4b63-f3ad-468a-a224-576f728e9e06" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417239 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417246 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebdb2c6-1b1f-4b97-b658-8f806387b52c" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417274 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a7644595-12fe-48ac-92c8-765cc57d4a4c" containerName="revoke-keys" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417282 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7644595-12fe-48ac-92c8-765cc57d4a4c" containerName="revoke-keys" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417295 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:36.417344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417302 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417463 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="874f4254-ae13-4ec0-8afb-c4342ca765ed" containerName="cleanup" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417472 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b153d4-a17d-4cd2-ab3a-308be9b516f9" containerName="cleanup" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417483 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a7a920-701a-4c4f-bc06-bb73f00573e1" containerName="cleanup" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417492 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1538b70d-4c8e-4226-afc6-5a4bcaa6836d" containerName="cleanup" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417498 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57a32dc-2a05-4f72-8a8c-aeb14c9ab07b" containerName="cleanup" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417510 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7644595-12fe-48ac-92c8-765cc57d4a4c" containerName="revoke-keys" Nov 01 15:15:36.417707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.417516 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="24545ec6-f2fa-4658-9287-e915c228e8e0" containerName="cleanup" Nov 01 15:15:36.422642 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.422623 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.471699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.471672 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt"] Nov 01 15:15:36.517966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.517940 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:15:36.518134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.518014 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:15:36.587120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.587088 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9bc\" (UniqueName: \"kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.587303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.587138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.587303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.587232 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.688736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.688650 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9bc\" (UniqueName: \"kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.688736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.688695 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.688736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.688729 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.689479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.689455 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.691008 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.690984 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:36.745922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:36.745898 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9bc\" (UniqueName: \"kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc\") pod \"maas-api-revoke-sub-0bd2db9c-5sbmt\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:37.037102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:37.037070 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:37.415990 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:15:37.415960 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bd3d1b_b9f4_4dd3_953c_ce2517277cfe.slice/crio-710642587c3a624da2452c4f79ae10602f83b9d82381c4ba99b102745a260298 WatchSource:0}: Error finding container 710642587c3a624da2452c4f79ae10602f83b9d82381c4ba99b102745a260298: Status 404 returned error can't find the container with id 710642587c3a624da2452c4f79ae10602f83b9d82381c4ba99b102745a260298 Nov 01 15:15:37.419581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:37.419555 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt"] Nov 01 15:15:37.765551 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:37.765521 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" event={"ID":"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe","Type":"ContainerStarted","Data":"be356586e792ef4aa9a0b433829f81e35e5cf3ac10009b4ea48cd630ccd9d9b4"} Nov 01 15:15:37.765682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:37.765581 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" event={"ID":"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe","Type":"ContainerStarted","Data":"710642587c3a624da2452c4f79ae10602f83b9d82381c4ba99b102745a260298"} Nov 01 15:15:37.880438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:37.880387 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" podStartSLOduration=1.880373477 podStartE2EDuration="1.880373477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:15:37.878597621 +0000 UTC m=+3476.936749471" watchObservedRunningTime="2026-11-01 15:15:37.880373477 +0000 UTC m=+3476.938525327" Nov 01 15:15:38.784829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:38.784789 5473 generic.go:356] "Generic (PLEG): container finished" podID="b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" containerID="be356586e792ef4aa9a0b433829f81e35e5cf3ac10009b4ea48cd630ccd9d9b4" exitCode=0 Nov 01 15:15:38.785263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:38.784875 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" event={"ID":"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe","Type":"ContainerDied","Data":"be356586e792ef4aa9a0b433829f81e35e5cf3ac10009b4ea48cd630ccd9d9b4"} Nov 01 15:15:39.009866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:39.009837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:39.947633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:39.947611 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:40.135482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.135404 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca\") pod \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " Nov 01 15:15:40.135648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.135629 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca\") pod \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " Nov 01 15:15:40.135698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.135672 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc\" (UniqueName: \"kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc\") pod \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\" (UID: \"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe\") " Nov 01 15:15:40.136012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.135971 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca" pod "b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" (UID: "b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:40.136270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.136246 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:40.137754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.137727 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca" pod "b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" (UID: "b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:15:40.137870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.137852 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc" pod "b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" (UID: "b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe"). InnerVolumeSpecName "kube-api-access-cz9bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:40.237789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.237755 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz9bc\" (UniqueName: \"kubernetes.io/projected/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-kube-api-access-cz9bc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:40.237789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.237784 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:40.802813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.802781 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" Nov 01 15:15:40.802984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.802833 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt" event={"ID":"b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe","Type":"ContainerDied","Data":"710642587c3a624da2452c4f79ae10602f83b9d82381c4ba99b102745a260298"} Nov 01 15:15:40.802984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:40.802881 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="710642587c3a624da2452c4f79ae10602f83b9d82381c4ba99b102745a260298" Nov 01 15:15:43.444113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:43.444035 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:43.742179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:43.742085 5473 scope.go:122] "RemoveContainer" containerID="68c72189d240088d87f63db8c38bb3615ecd0f5e4485cb46961fe6bf888af205" Nov 01 15:15:43.752948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:43.752931 5473 scope.go:122] "RemoveContainer" containerID="0f191dc3e73c682628ef5a9447b66ea5761f2432893d3b11c04a49cd067d6c6b" Nov 01 15:15:45.758460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:45.758427 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:48.221265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:48.221206 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:50.210377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:50.210338 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5"] Nov 01 15:15:50.334239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:50.317657 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:51.716244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:51.716186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:51.750884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:51.750849 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n"] Nov 01 15:15:51.859880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:51.859844 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9"] Nov 01 15:15:52.160987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:52.160953 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:52.313109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:52.313076 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:53.043208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:53.043172 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:56.752152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:56.752094 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" podUID="6ab1f1be-9eb6-4056-9012-711080246644" containerName="payload-processing" containerID="cri-o://27f0037d9bca4f80431b11a355ffaa4e14656ee6f6feb798d2b3978300b17ff1" gracePeriod=25 Nov 01 15:15:56.860691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:56.860651 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" podUID="59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" containerName="payload-pre-processing" containerID="cri-o://b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed" gracePeriod=25 Nov 01 15:15:56.931357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:56.931256 5473 generic.go:356] "Generic (PLEG): container finished" podID="6ab1f1be-9eb6-4056-9012-711080246644" containerID="27f0037d9bca4f80431b11a355ffaa4e14656ee6f6feb798d2b3978300b17ff1" exitCode=0 Nov 01 15:15:56.931357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:56.931354 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" event={"ID":"6ab1f1be-9eb6-4056-9012-711080246644","Type":"ContainerDied","Data":"27f0037d9bca4f80431b11a355ffaa4e14656ee6f6feb798d2b3978300b17ff1"} Nov 01 15:15:57.029930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.029904 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:15:57.040513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.040480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:57.126063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.126037 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2\" (UniqueName: \"kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2\") pod \"6ab1f1be-9eb6-4056-9012-711080246644\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " Nov 01 15:15:57.126240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.126115 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume\") pod \"6ab1f1be-9eb6-4056-9012-711080246644\" (UID: \"6ab1f1be-9eb6-4056-9012-711080246644\") " Nov 01 15:15:57.126531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.126502 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume" pod "6ab1f1be-9eb6-4056-9012-711080246644" (UID: "6ab1f1be-9eb6-4056-9012-711080246644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:57.126824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.126805 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab1f1be-9eb6-4056-9012-711080246644-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:57.128261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.128209 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2" pod "6ab1f1be-9eb6-4056-9012-711080246644" (UID: "6ab1f1be-9eb6-4056-9012-711080246644"). InnerVolumeSpecName "kube-api-access-mghb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:57.147382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.147351 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:15:57.228076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.228043 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume\") pod \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " Nov 01 15:15:57.228291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.228092 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt\" (UniqueName: \"kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt\") pod \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\" (UID: \"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b\") " Nov 01 15:15:57.228468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.228451 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mghb2\" (UniqueName: \"kubernetes.io/projected/6ab1f1be-9eb6-4056-9012-711080246644-kube-api-access-mghb2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:57.228571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.228536 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume" pod "59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" (UID: "59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:15:57.230339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.230317 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt" pod "59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" (UID: "59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b"). InnerVolumeSpecName "kube-api-access-d6vbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:15:57.258421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.258340 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:57.330204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.330165 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:57.330204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.330201 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vbt\" (UniqueName: \"kubernetes.io/projected/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b-kube-api-access-d6vbt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:15:57.642869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.642834 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:15:57.940529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.940441 5473 generic.go:356] "Generic (PLEG): container finished" podID="59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" containerID="b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed" exitCode=0 Nov 01 15:15:57.940529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.940496 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" Nov 01 15:15:57.941060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.940530 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" event={"ID":"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b","Type":"ContainerDied","Data":"b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed"} Nov 01 15:15:57.941060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.940598 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9" event={"ID":"59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b","Type":"ContainerDied","Data":"2e68086d5ae46b7f05114c8bff3dd128d5448c0ffb08f8e10e10b506637882ac"} Nov 01 15:15:57.941060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.940634 5473 scope.go:122] "RemoveContainer" containerID="b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed" Nov 01 15:15:57.942209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.942190 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" Nov 01 15:15:57.942209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.942205 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n" event={"ID":"6ab1f1be-9eb6-4056-9012-711080246644","Type":"ContainerDied","Data":"413ca216a549e691188152eb5a1174611d5317c232589d0c4dd1571929b44085"} Nov 01 15:15:57.952799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.952779 5473 scope.go:122] "RemoveContainer" containerID="b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed" Nov 01 15:15:57.953129 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:15:57.953102 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed\": container with ID starting with b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed not found: ID does not exist" containerID="b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed" Nov 01 15:15:57.953207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.953138 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed"} err="failed to get container status \"b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed\": rpc error: code = NotFound desc = could not find container \"b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed\": container with ID starting with b7a07b736b176c9a697cc40922826f53b2c66a7e2fa6f69f9e4747cf6ec2f7ed not found: ID does not exist" Nov 01 15:15:57.953207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:57.953158 5473 scope.go:122] "RemoveContainer" containerID="27f0037d9bca4f80431b11a355ffaa4e14656ee6f6feb798d2b3978300b17ff1" Nov 01 15:15:58.105362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:58.105326 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9"] Nov 01 15:15:58.109110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:58.109083 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0f1e68-84755db8b7-cgld9"] Nov 01 15:15:58.220592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:58.220518 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n"] Nov 01 15:15:58.228993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:58.228959 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0f1e68-668456c4f5-hgl6n"] Nov 01 15:15:59.082701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:59.082670 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" path="/var/lib/kubelet/pods/59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b/volumes" Nov 01 15:15:59.083116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:15:59.083071 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab1f1be-9eb6-4056-9012-711080246644" path="/var/lib/kubelet/pods/6ab1f1be-9eb6-4056-9012-711080246644/volumes" Nov 01 15:16:01.108852 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:16:01.108817 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:16:01.583573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:01.583536 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:01.968758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:01.968681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:02.137188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:02.137156 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:02.299904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:02.299872 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:03.366895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:03.366862 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:03.796766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:03.796728 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv"] Nov 01 15:16:04.155924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:04.155827 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:04.321267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:04.321164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:05.231354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.231314 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" podUID="83ca903f-828b-47b9-b2ac-7b8cc40332b1" containerName="main" containerID="cri-o://ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747" gracePeriod=45 Nov 01 15:16:05.535574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.535545 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:16:05.657269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657233 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs\") pod \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " Nov 01 15:16:05.657485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657368 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home\") pod \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " Nov 01 15:16:05.657485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657419 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx\" (UniqueName: \"kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx\") pod \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " Nov 01 15:16:05.657485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657471 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir\") pod \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " Nov 01 15:16:05.657672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657576 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm\") pod \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " Nov 01 15:16:05.657672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657637 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache\") pod \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\" (UID: \"83ca903f-828b-47b9-b2ac-7b8cc40332b1\") " Nov 01 15:16:05.657835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657669 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home" pod "83ca903f-828b-47b9-b2ac-7b8cc40332b1" (UID: "83ca903f-828b-47b9-b2ac-7b8cc40332b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:05.657900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.657874 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir" pod "83ca903f-828b-47b9-b2ac-7b8cc40332b1" (UID: "83ca903f-828b-47b9-b2ac-7b8cc40332b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:05.658089 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.658065 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache" pod "83ca903f-828b-47b9-b2ac-7b8cc40332b1" (UID: "83ca903f-828b-47b9-b2ac-7b8cc40332b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:05.658380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.658359 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:05.658533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.658518 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:05.665208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.665181 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs" pod "83ca903f-828b-47b9-b2ac-7b8cc40332b1" (UID: "83ca903f-828b-47b9-b2ac-7b8cc40332b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:16:05.665208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.665187 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm" pod "83ca903f-828b-47b9-b2ac-7b8cc40332b1" (UID: "83ca903f-828b-47b9-b2ac-7b8cc40332b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:05.673345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.673316 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx" pod "83ca903f-828b-47b9-b2ac-7b8cc40332b1" (UID: "83ca903f-828b-47b9-b2ac-7b8cc40332b1"). InnerVolumeSpecName "kube-api-access-47jpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:16:05.759630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.759547 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47jpx\" (UniqueName: \"kubernetes.io/projected/83ca903f-828b-47b9-b2ac-7b8cc40332b1-kube-api-access-47jpx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:05.759630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.759579 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:05.759630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.759590 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83ca903f-828b-47b9-b2ac-7b8cc40332b1-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:05.759630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:05.759600 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83ca903f-828b-47b9-b2ac-7b8cc40332b1-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:06.006506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.006468 5473 generic.go:356] "Generic (PLEG): container finished" podID="83ca903f-828b-47b9-b2ac-7b8cc40332b1" containerID="ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747" exitCode=0 Nov 01 15:16:06.006506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.006513 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" Nov 01 15:16:06.006760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.006552 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" event={"ID":"83ca903f-828b-47b9-b2ac-7b8cc40332b1","Type":"ContainerDied","Data":"ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747"} Nov 01 15:16:06.006760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.006599 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5" event={"ID":"83ca903f-828b-47b9-b2ac-7b8cc40332b1","Type":"ContainerDied","Data":"59d6d767c2d92ba3ffe6031dd517ade7a5cdcb1fe3b6e0a7fa374dc410d4759b"} Nov 01 15:16:06.006760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.006623 5473 scope.go:122] "RemoveContainer" containerID="ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747" Nov 01 15:16:06.017991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.017971 5473 scope.go:122] "RemoveContainer" containerID="ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747" Nov 01 15:16:06.018324 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:16:06.018302 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747\": container with ID starting with ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747 not found: ID does not exist" containerID="ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747" Nov 01 15:16:06.018389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.018333 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747"} err="failed to get container status \"ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747\": rpc error: code = NotFound desc = could not find container \"ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747\": container with ID starting with ea73340f5aab5e5cddf3ac0f160043903ea95b11e40947c93a0e3c50317f8747 not found: ID does not exist" Nov 01 15:16:06.172618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.172580 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5"] Nov 01 15:16:06.176068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.176039 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/auth-test-model-1ac11c-kserve-f7dcdfd69-7j9d5"] Nov 01 15:16:06.775394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:06.775359 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:07.084084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:07.083988 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ca903f-828b-47b9-b2ac-7b8cc40332b1" path="/var/lib/kubelet/pods/83ca903f-828b-47b9-b2ac-7b8cc40332b1/volumes" Nov 01 15:16:08.797532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:08.797485 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" podUID="f5540bee-3f36-487a-bd82-dbcf44e15c2b" containerName="payload-processing" containerID="cri-o://b38b16b206a3d9a1766ddc368cf838c6167a8aa783ba757830cb98a08ed210fa" gracePeriod=25 Nov 01 15:16:09.037632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.037603 5473 generic.go:356] "Generic (PLEG): container finished" podID="f5540bee-3f36-487a-bd82-dbcf44e15c2b" containerID="b38b16b206a3d9a1766ddc368cf838c6167a8aa783ba757830cb98a08ed210fa" exitCode=0 Nov 01 15:16:09.037784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.037684 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" event={"ID":"f5540bee-3f36-487a-bd82-dbcf44e15c2b","Type":"ContainerDied","Data":"b38b16b206a3d9a1766ddc368cf838c6167a8aa783ba757830cb98a08ed210fa"} Nov 01 15:16:09.067177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.067111 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:16:09.102444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.102412 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84\" (UniqueName: \"kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84\") pod \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " Nov 01 15:16:09.102601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.102526 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume\") pod \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\" (UID: \"f5540bee-3f36-487a-bd82-dbcf44e15c2b\") " Nov 01 15:16:09.102958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.102932 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume" pod "f5540bee-3f36-487a-bd82-dbcf44e15c2b" (UID: "f5540bee-3f36-487a-bd82-dbcf44e15c2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:16:09.104526 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.104501 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84" pod "f5540bee-3f36-487a-bd82-dbcf44e15c2b" (UID: "f5540bee-3f36-487a-bd82-dbcf44e15c2b"). InnerVolumeSpecName "kube-api-access-4ff84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:16:09.136185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.136151 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:09.204354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.204320 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ff84\" (UniqueName: \"kubernetes.io/projected/f5540bee-3f36-487a-bd82-dbcf44e15c2b-kube-api-access-4ff84\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:09.204354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:09.204351 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5540bee-3f36-487a-bd82-dbcf44e15c2b-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:10.046197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:10.046161 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" Nov 01 15:16:10.046713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:10.046199 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv" event={"ID":"f5540bee-3f36-487a-bd82-dbcf44e15c2b","Type":"ContainerDied","Data":"f7332d6ed70fd8b168f24682ec847f09afd2e5b3fd50431d6ccf8952dab83a8e"} Nov 01 15:16:10.046713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:10.046305 5473 scope.go:122] "RemoveContainer" containerID="b38b16b206a3d9a1766ddc368cf838c6167a8aa783ba757830cb98a08ed210fa" Nov 01 15:16:10.212582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:10.212551 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv"] Nov 01 15:16:10.262394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:10.262360 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2cf1ee-6659d86bb7-cpspv"] Nov 01 15:16:11.083110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.083079 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5540bee-3f36-487a-bd82-dbcf44e15c2b" path="/var/lib/kubelet/pods/f5540bee-3f36-487a-bd82-dbcf44e15c2b/volumes" Nov 01 15:16:11.156743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.156710 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:11.263597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.263562 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh"] Nov 01 15:16:11.343563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.343478 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:11.346809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.346785 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7387-7b3abb"] Nov 01 15:16:11.347435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347421 5473 cpu_manager.go:414] "containerMap: removing container" podUID="83ca903f-828b-47b9-b2ac-7b8cc40332b1" containerName="main" Nov 01 15:16:11.347483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347436 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ca903f-828b-47b9-b2ac-7b8cc40332b1" containerName="main" Nov 01 15:16:11.347483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347450 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f5540bee-3f36-487a-bd82-dbcf44e15c2b" containerName="payload-processing" Nov 01 15:16:11.347483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347455 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5540bee-3f36-487a-bd82-dbcf44e15c2b" containerName="payload-processing" Nov 01 15:16:11.347483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347478 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6ab1f1be-9eb6-4056-9012-711080246644" containerName="payload-processing" Nov 01 15:16:11.347483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347483 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab1f1be-9eb6-4056-9012-711080246644" containerName="payload-processing" Nov 01 15:16:11.347662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347492 5473 cpu_manager.go:414] "containerMap: removing container" podUID="59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" containerName="payload-pre-processing" Nov 01 15:16:11.347662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347497 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" containerName="payload-pre-processing" Nov 01 15:16:11.347662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347513 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" containerName="revoke-keys" Nov 01 15:16:11.347662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347518 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" containerName="revoke-keys" Nov 01 15:16:11.347853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347739 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ca903f-828b-47b9-b2ac-7b8cc40332b1" containerName="main" Nov 01 15:16:11.347853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347757 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5540bee-3f36-487a-bd82-dbcf44e15c2b" containerName="payload-processing" Nov 01 15:16:11.347853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347774 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab1f1be-9eb6-4056-9012-711080246644" containerName="payload-processing" Nov 01 15:16:11.347853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347784 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="59fb9ec5-b4cb-4969-93fc-5e7d0d4d9e8b" containerName="payload-pre-processing" Nov 01 15:16:11.347853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.347795 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" containerName="revoke-keys" Nov 01 15:16:11.359828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.359793 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:11.420315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.420278 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-7b3abb"] Nov 01 15:16:11.435910 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.435875 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj8xh\" (UniqueName: \"kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh\") pod \"test-curl-7387-7b3abb\" (UID: \"2d664c62-242e-4cc8-b7ed-7ad3ef149763\") " pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:11.441495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.441462 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jngw\"" Nov 01 15:16:11.538323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.538286 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj8xh\" (UniqueName: \"kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh\") pod \"test-curl-7387-7b3abb\" (UID: \"2d664c62-242e-4cc8-b7ed-7ad3ef149763\") " pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:11.595248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.595155 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj8xh\" (UniqueName: \"kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh\") pod \"test-curl-7387-7b3abb\" (UID: \"2d664c62-242e-4cc8-b7ed-7ad3ef149763\") " pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:11.721996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:11.721956 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:12.103150 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:16:12.097369 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d664c62_242e_4cc8_b7ed_7ad3ef149763.slice/crio-35c730cfedbef1c50e0e162abfe96c4a2d1889e9e50b7257ee9d8b242e71566e WatchSource:0}: Error finding container 35c730cfedbef1c50e0e162abfe96c4a2d1889e9e50b7257ee9d8b242e71566e: Status 404 returned error can't find the container with id 35c730cfedbef1c50e0e162abfe96c4a2d1889e9e50b7257ee9d8b242e71566e Nov 01 15:16:12.103150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:12.101451 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7387-7b3abb"] Nov 01 15:16:13.073303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:13.073268 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-7b3abb" event={"ID":"2d664c62-242e-4cc8-b7ed-7ad3ef149763","Type":"ContainerStarted","Data":"91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b"} Nov 01 15:16:13.073510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:13.073312 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7387-7b3abb" event={"ID":"2d664c62-242e-4cc8-b7ed-7ad3ef149763","Type":"ContainerStarted","Data":"35c730cfedbef1c50e0e162abfe96c4a2d1889e9e50b7257ee9d8b242e71566e"} Nov 01 15:16:13.192116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:13.192061 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7387-7b3abb" podStartSLOduration=2.192047459 podStartE2EDuration="2.192047459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:16:13.189199521 +0000 UTC m=+3512.247351370" watchObservedRunningTime="2026-11-01 15:16:13.192047459 +0000 UTC m=+3512.250199309" Nov 01 15:16:14.697425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:14.697389 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7387-7b3abb"] Nov 01 15:16:14.700754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:14.700726 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7387-7b3abb"] Nov 01 15:16:15.086897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:15.086859 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7387-7b3abb" podUID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" containerName="test-curl-7387-7b3abb" containerID="cri-o://91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b" gracePeriod=2 Nov 01 15:16:15.712131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:15.712089 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:16.177448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.177411 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:16.732984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.732944 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8"] Nov 01 15:16:16.735338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.735307 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" containerName="test-curl-7387-7b3abb" Nov 01 15:16:16.735529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.735504 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" containerName="test-curl-7387-7b3abb" Nov 01 15:16:16.735795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.735776 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" containerName="test-curl-7387-7b3abb" Nov 01 15:16:16.741393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.741363 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771672 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlh9\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771719 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771740 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771757 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771806 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771832 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.771891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771888 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.772310 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771918 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.772310 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.771991 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.791924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.791887 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8"] Nov 01 15:16:16.798610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.798581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:16.858108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.858079 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-55943a-openshift-default-dockercfg-pbw6d\"" Nov 01 15:16:16.872980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.872948 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlh9\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.872998 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873111 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873149 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873272 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873314 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873377 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873489 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873543 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873670 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873669 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873873 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873941 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.873997 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.873975 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.875802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.875782 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.875906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.875778 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.931921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.931889 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:16.932085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:16.932001 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlh9\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9\") pod \"e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:17.032998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.032966 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:17.075295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.075257 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:17.333289 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.333262 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:17.380897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.380863 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh\" (UniqueName: \"kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh\") pod \"2d664c62-242e-4cc8-b7ed-7ad3ef149763\" (UID: \"2d664c62-242e-4cc8-b7ed-7ad3ef149763\") " Nov 01 15:16:17.383068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.383038 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh" pod "2d664c62-242e-4cc8-b7ed-7ad3ef149763" (UID: "2d664c62-242e-4cc8-b7ed-7ad3ef149763"). InnerVolumeSpecName "kube-api-access-kj8xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:16:17.483090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.483048 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj8xh\" (UniqueName: \"kubernetes.io/projected/2d664c62-242e-4cc8-b7ed-7ad3ef149763-kube-api-access-kj8xh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:17.509617 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:16:17.509571 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef29d51_4057_4f6c_8533_c03c355a53db.slice/crio-a23b0ddfd34da4053411c42fe087ae4f98ff041dbe4b2a1870663730b44bfeba WatchSource:0}: Error finding container a23b0ddfd34da4053411c42fe087ae4f98ff041dbe4b2a1870663730b44bfeba: Status 404 returned error can't find the container with id a23b0ddfd34da4053411c42fe087ae4f98ff041dbe4b2a1870663730b44bfeba Nov 01 15:16:17.512561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.512536 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8"] Nov 01 15:16:17.512878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.512844 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:16:17.512945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.512928 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:16:17.512989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:17.512975 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:16:18.110760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.110729 5473 generic.go:356] "Generic (PLEG): container finished" podID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" containerID="91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b" exitCode=137 Nov 01 15:16:18.111209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.110771 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:18.111209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.110798 5473 scope.go:122] "RemoveContainer" containerID="91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b" Nov 01 15:16:18.112749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.112723 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" event={"ID":"6ef29d51-4057-4f6c-8533-c03c355a53db","Type":"ContainerStarted","Data":"3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006"} Nov 01 15:16:18.112877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.112776 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" event={"ID":"6ef29d51-4057-4f6c-8533-c03c355a53db","Type":"ContainerStarted","Data":"a23b0ddfd34da4053411c42fe087ae4f98ff041dbe4b2a1870663730b44bfeba"} Nov 01 15:16:18.124632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.124352 5473 scope.go:122] "RemoveContainer" containerID="91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b" Nov 01 15:16:18.125334 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:16:18.125310 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b\": container with ID starting with 91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b not found: ID does not exist" containerID="91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b" Nov 01 15:16:18.125428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.125346 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b"} err="failed to get container status \"91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b\": rpc error: code = NotFound desc = could not find container \"91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b\": container with ID starting with 91805a5b462e20f9f9b00eef039541745b779ef2228f4789d47606dfbb59e63b not found: ID does not exist" Nov 01 15:16:18.233683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.233580 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podStartSLOduration=2.233565041 podStartE2EDuration="2.233565041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:16:18.229365121 +0000 UTC m=+3517.287516971" watchObservedRunningTime="2026-11-01 15:16:18.233565041 +0000 UTC m=+3517.291716964" Nov 01 15:16:18.280623 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:16:18.280585 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7387-7b3abb\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" pod="opendatahub/test-curl-7387-7b3abb" Nov 01 15:16:18.437103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:18.437067 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:19.083630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:19.083600 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d664c62-242e-4cc8-b7ed-7ad3ef149763" path="/var/lib/kubelet/pods/2d664c62-242e-4cc8-b7ed-7ad3ef149763/volumes" Nov 01 15:16:19.083947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:19.083931 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:19.084034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:19.084019 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:19.121363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:19.121328 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:19.122320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:19.122303 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:16:20.248571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:20.248527 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:21.069962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:21.069923 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:21.251595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:21.251542 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:24.181270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.181204 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:24.348149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.348113 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:24.566742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.566708 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9"] Nov 01 15:16:24.572486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.572466 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.635944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.635908 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9"] Nov 01 15:16:24.678542 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.678508 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-55943a\"" Nov 01 15:16:24.678959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.678940 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-55943a-dockercfg-ws48d\"" Nov 01 15:16:24.680097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.680070 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.680494 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.680475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f8km\" (UniqueName: \"kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km\") pod \"payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.734973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.734931 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28"] Nov 01 15:16:24.741554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.741490 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:24.782335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.782305 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f8km\" (UniqueName: \"kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km\") pod \"payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.782481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.782447 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.783148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.783126 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.783897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.783876 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28"] Nov 01 15:16:24.839679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.839601 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f8km\" (UniqueName: \"kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km\") pod \"payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.884007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.883959 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:24.884007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.884006 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz97k\" (UniqueName: \"kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k\") pod \"payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:24.886971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.886952 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:24.985335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.985299 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:24.985335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.985337 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz97k\" (UniqueName: \"kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k\") pod \"payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:24.986117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:24.986088 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:25.030840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.030785 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:25.044246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.044183 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz97k\" (UniqueName: \"kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k\") pod \"payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:25.059036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.059007 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:25.205443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.205377 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:25.259034 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:16:25.258993 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1bef25_851f_4c0a_86eb_bbc95f60d528.slice/crio-871e7db5ed5b023a470d8cedbacf69685995f724543e60bda746abd7e4d2699c WatchSource:0}: Error finding container 871e7db5ed5b023a470d8cedbacf69685995f724543e60bda746abd7e4d2699c: Status 404 returned error can't find the container with id 871e7db5ed5b023a470d8cedbacf69685995f724543e60bda746abd7e4d2699c Nov 01 15:16:25.262899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.262871 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9"] Nov 01 15:16:25.430152 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:16:25.430111 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a9e3ac_dcad_4ec3_8e8d_e7956314f3fb.slice/crio-115817ac6f5bbb9b984c040c0fda30e856df389e49f9f0e1ac79e43ed0dace24 WatchSource:0}: Error finding container 115817ac6f5bbb9b984c040c0fda30e856df389e49f9f0e1ac79e43ed0dace24: Status 404 returned error can't find the container with id 115817ac6f5bbb9b984c040c0fda30e856df389e49f9f0e1ac79e43ed0dace24 Nov 01 15:16:25.435473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.435442 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28"] Nov 01 15:16:25.637115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.637080 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:25.808891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:25.808853 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:26.176058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.176026 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" event={"ID":"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb","Type":"ContainerStarted","Data":"87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688"} Nov 01 15:16:26.176244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.176080 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" event={"ID":"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb","Type":"ContainerStarted","Data":"115817ac6f5bbb9b984c040c0fda30e856df389e49f9f0e1ac79e43ed0dace24"} Nov 01 15:16:26.176244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.176142 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:26.177788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.177767 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" event={"ID":"0d1bef25-851f-4c0a-86eb-bbc95f60d528","Type":"ContainerStarted","Data":"d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6"} Nov 01 15:16:26.177917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.177800 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" event={"ID":"0d1bef25-851f-4c0a-86eb-bbc95f60d528","Type":"ContainerStarted","Data":"871e7db5ed5b023a470d8cedbacf69685995f724543e60bda746abd7e4d2699c"} Nov 01 15:16:26.177966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.177910 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:26.284833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.284798 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" podUID="0d3fe657-30d8-45b6-82f0-91b3532227ea" containerName="main" containerID="cri-o://cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d" gracePeriod=45 Nov 01 15:16:26.295357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.295305 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" podStartSLOduration=2.295290198 podStartE2EDuration="2.295290198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:16:26.290771743 +0000 UTC m=+3525.348923594" watchObservedRunningTime="2026-11-01 15:16:26.295290198 +0000 UTC m=+3525.353442103" Nov 01 15:16:26.637863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.637521 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:16:26.711481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711391 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir\") pod \"0d3fe657-30d8-45b6-82f0-91b3532227ea\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " Nov 01 15:16:26.711481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711430 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm\") pod \"0d3fe657-30d8-45b6-82f0-91b3532227ea\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " Nov 01 15:16:26.711737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711564 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home\") pod \"0d3fe657-30d8-45b6-82f0-91b3532227ea\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " Nov 01 15:16:26.711737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711585 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache\") pod \"0d3fe657-30d8-45b6-82f0-91b3532227ea\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " Nov 01 15:16:26.711737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711654 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs\") pod \"0d3fe657-30d8-45b6-82f0-91b3532227ea\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " Nov 01 15:16:26.711737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711656 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir" pod "0d3fe657-30d8-45b6-82f0-91b3532227ea" (UID: "0d3fe657-30d8-45b6-82f0-91b3532227ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:26.711737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711738 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5\" (UniqueName: \"kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5\") pod \"0d3fe657-30d8-45b6-82f0-91b3532227ea\" (UID: \"0d3fe657-30d8-45b6-82f0-91b3532227ea\") " Nov 01 15:16:26.711991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711803 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache" pod "0d3fe657-30d8-45b6-82f0-91b3532227ea" (UID: "0d3fe657-30d8-45b6-82f0-91b3532227ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:26.711991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.711820 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home" pod "0d3fe657-30d8-45b6-82f0-91b3532227ea" (UID: "0d3fe657-30d8-45b6-82f0-91b3532227ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:26.712259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.712238 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:26.712365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.712264 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:26.712365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.712278 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:26.713863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.713840 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm" pod "0d3fe657-30d8-45b6-82f0-91b3532227ea" (UID: "0d3fe657-30d8-45b6-82f0-91b3532227ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:16:26.713965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.713874 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5" pod "0d3fe657-30d8-45b6-82f0-91b3532227ea" (UID: "0d3fe657-30d8-45b6-82f0-91b3532227ea"). InnerVolumeSpecName "kube-api-access-gj2b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:16:26.713965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.713900 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs" pod "0d3fe657-30d8-45b6-82f0-91b3532227ea" (UID: "0d3fe657-30d8-45b6-82f0-91b3532227ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:16:26.753752 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.753689 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" podStartSLOduration=2.75367011 podStartE2EDuration="2.75367011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:16:26.418890921 +0000 UTC m=+3525.477042794" watchObservedRunningTime="2026-11-01 15:16:26.75367011 +0000 UTC m=+3525.811821962" Nov 01 15:16:26.814111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.814060 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3fe657-30d8-45b6-82f0-91b3532227ea-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:26.814111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.814107 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj2b5\" (UniqueName: \"kubernetes.io/projected/0d3fe657-30d8-45b6-82f0-91b3532227ea-kube-api-access-gj2b5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:26.814111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:26.814122 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d3fe657-30d8-45b6-82f0-91b3532227ea-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:16:27.186601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.186567 5473 generic.go:356] "Generic (PLEG): container finished" podID="0d3fe657-30d8-45b6-82f0-91b3532227ea" containerID="cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d" exitCode=0 Nov 01 15:16:27.186806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.186619 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" Nov 01 15:16:27.186806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.186648 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" event={"ID":"0d3fe657-30d8-45b6-82f0-91b3532227ea","Type":"ContainerDied","Data":"cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d"} Nov 01 15:16:27.186806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.186689 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh" event={"ID":"0d3fe657-30d8-45b6-82f0-91b3532227ea","Type":"ContainerDied","Data":"465a4085498fa76761eac0f50e283afbec530b5441dba2408634edeba9f583cf"} Nov 01 15:16:27.186806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.186707 5473 scope.go:122] "RemoveContainer" containerID="cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d" Nov 01 15:16:27.200756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.200734 5473 scope.go:122] "RemoveContainer" containerID="cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d" Nov 01 15:16:27.200995 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:16:27.200976 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d\": container with ID starting with cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d not found: ID does not exist" containerID="cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d" Nov 01 15:16:27.201070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.201003 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d"} err="failed to get container status \"cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d\": rpc error: code = NotFound desc = could not find container \"cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d\": container with ID starting with cbee1ac2ca47e304d8fd3b2f530ee56ae4bbdb6ed1cf92f9db471ef634db8c2d not found: ID does not exist" Nov 01 15:16:27.352255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.352151 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh"] Nov 01 15:16:27.355724 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:27.355695 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-8fd363/auth-test-model-8fd363-kserve-84dbf64bb9-6qqdh"] Nov 01 15:16:29.090460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:29.090425 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3fe657-30d8-45b6-82f0-91b3532227ea" path="/var/lib/kubelet/pods/0d3fe657-30d8-45b6-82f0-91b3532227ea/volumes" Nov 01 15:16:29.881792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:29.881739 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:30.343879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:30.343836 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:30.552799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:30.552756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:31.108378 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:16:31.108342 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:16:31.348898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:31.348864 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:34.787653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:34.787607 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:37.189998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:37.189968 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:16:37.190456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:37.190043 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:16:39.544512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:39.544478 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:43.328930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.328898 5473 generic.go:356] "Generic (PLEG): container finished" podID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerID="514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910" exitCode=0 Nov 01 15:16:43.329419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.328967 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" event={"ID":"6b65ea53-8ec4-4ac8-8063-755d1f64faec","Type":"ContainerDied","Data":"514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910"} Nov 01 15:16:43.329419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.329402 5473 scope.go:122] "RemoveContainer" containerID="514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910" Nov 01 15:16:43.331164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.331134 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:16:43.331275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.331208 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:16:43.331275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.331255 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:16:43.856347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.856315 5473 scope.go:122] "RemoveContainer" containerID="8dabdc63666c192e044102e28b2fe750da8634e168e62874c081f1fb83707459" Nov 01 15:16:43.867398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:43.867375 5473 scope.go:122] "RemoveContainer" containerID="08e4939ed654accabceba68ec248f4d56eeabf7942bcadb1376f5be1d2e35547" Nov 01 15:16:44.358106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:44.358068 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" event={"ID":"6b65ea53-8ec4-4ac8-8063-755d1f64faec","Type":"ContainerStarted","Data":"4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9"} Nov 01 15:16:44.389864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:44.389831 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:44.718458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:44.718363 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:16:44.718678 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:44.718654 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 15:16:44.718775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:44.718713 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 15:16:45.719057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:45.719020 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 15:16:45.719522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:45.719074 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 15:16:46.718586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:46.718544 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 15:16:46.718797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:46.718601 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 15:16:47.719052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:47.719010 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 15:16:47.719507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:47.719079 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 15:16:48.718488 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:48.718449 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 01 15:16:48.718688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:48.718507 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 01 15:16:48.895070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:48.895031 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:49.723689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:49.723652 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:16:49.723942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:49.723924 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:16:49.724779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:49.724759 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:16:53.310434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:53.310403 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:16:57.738789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:16:57.738754 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:01.108679 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:01.108643 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:17:01.896883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:01.896852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:03.853818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:03.853786 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:06.056936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:06.056898 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:07.811156 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:07.811122 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:12.203094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:12.203059 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:14.259442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:14.259411 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:14.422691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:14.422657 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:15.809524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:15.809490 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:16.253094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:16.253017 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx"] Nov 01 15:17:16.253275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:16.253251 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" containerID="cri-o://0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02" gracePeriod=30 Nov 01 15:17:16.268321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:16.268292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:16.797586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:16.797547 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:16.797983 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:16.797961 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:17:16.798078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:16.798013 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:17:17.010700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:17.010663 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:20.732679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:20.732646 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:21.610729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.610700 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:17:21.641245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.641184 5473 generic.go:356] "Generic (PLEG): container finished" podID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerID="0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02" exitCode=0 Nov 01 15:17:21.641245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.641248 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" Nov 01 15:17:21.641452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.641251 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" event={"ID":"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a","Type":"ContainerDied","Data":"0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02"} Nov 01 15:17:21.641452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.641309 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx" event={"ID":"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a","Type":"ContainerDied","Data":"9790a69463cf583955c902ecec1c7e1967cfd30f82c47603471a8f7ad1ad9fdb"} Nov 01 15:17:21.641452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.641334 5473 scope.go:122] "RemoveContainer" containerID="0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02" Nov 01 15:17:21.651916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.651897 5473 scope.go:122] "RemoveContainer" containerID="9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339" Nov 01 15:17:21.661700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.661683 5473 scope.go:122] "RemoveContainer" containerID="0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02" Nov 01 15:17:21.661969 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:21.661949 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02\": container with ID starting with 0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02 not found: ID does not exist" containerID="0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02" Nov 01 15:17:21.662009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.661978 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02"} err="failed to get container status \"0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02\": rpc error: code = NotFound desc = could not find container \"0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02\": container with ID starting with 0c6c32337adc5b3d2183ad4081ac10baa5092bcebc154a5706572fb587bc1e02 not found: ID does not exist" Nov 01 15:17:21.662009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.661996 5473 scope.go:122] "RemoveContainer" containerID="9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339" Nov 01 15:17:21.662243 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:21.662209 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339\": container with ID starting with 9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339 not found: ID does not exist" containerID="9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339" Nov 01 15:17:21.662290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.662250 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339"} err="failed to get container status \"9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339\": rpc error: code = NotFound desc = could not find container \"9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339\": container with ID starting with 9c9163165e35aafea61c5ff29c2d6f4bad378651a783491e88ebe680fc02d339 not found: ID does not exist" Nov 01 15:17:21.712615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712585 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.712788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712642 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.712788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712709 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.712788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712732 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.712788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712776 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.713103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712798 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.713103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712868 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:21.713103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712915 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.713103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.712984 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.713103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713022 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs\") pod \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\" (UID: \"1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a\") " Nov 01 15:17:21.713433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713187 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:21.713433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713228 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:17:21.713433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713375 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:21.713603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713543 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:21.713716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713696 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.713760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713723 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.713760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713738 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.713760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713752 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.713891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.713766 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.715030 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.715002 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:17:21.715261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.715238 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "kube-api-access-7bcjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:21.715492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.715469 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:21.715571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.715533 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token" pod "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" (UID: "1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:21.815960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.815371 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.815960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.815405 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.815960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.815421 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bcjh\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-kube-api-access-7bcjh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:21.815960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:21.815434 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:22.064321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:22.064278 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx"] Nov 01 15:17:22.067867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:22.067839 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0f1e68-openshift-default-568d4bc49c-w2sfx"] Nov 01 15:17:22.346813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:22.346775 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:22.917815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:22.917784 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:23.082771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:23.082734 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" path="/var/lib/kubelet/pods/1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a/volumes" Nov 01 15:17:23.911183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:23.911146 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:24.069854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:24.069822 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:25.561828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:25.561791 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:25.726734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:25.726701 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:27.022438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:27.022403 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:27.191615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:27.191584 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:27.624805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:27.624770 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg"] Nov 01 15:17:27.625017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:27.624994 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" containerID="cri-o://043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896" gracePeriod=30 Nov 01 15:17:27.657990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:27.657955 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:28.371518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:28.371487 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:29.192473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:29.192439 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:29.403780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:29.403748 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:31.109285 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:31.109249 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:17:31.746774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:31.746741 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:31.827253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:31.827196 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:17:31.827425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:31.827285 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:17:32.898547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:32.898486 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:17:33.062022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.061982 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062090 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062169 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062242 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062487 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062533 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:17:33.062761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062713 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:33.062893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062760 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062771 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:33.062893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062827 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.062893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062866 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.063138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.062933 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data\") pod \"cce384e4-cde3-4258-9484-45bcd3a10cea\" (UID: \"cce384e4-cde3-4258-9484-45bcd3a10cea\") " Nov 01 15:17:33.063673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.063650 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce384e4-cde3-4258-9484-45bcd3a10cea-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.063815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.063789 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:33.063933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.063915 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.064050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.064035 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.064202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.064174 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:33.065031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.065009 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:17:33.066111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.066086 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:33.066205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.066109 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "kube-api-access-pmn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:33.066367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.066349 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo" pod "cce384e4-cde3-4258-9484-45bcd3a10cea" (UID: "cce384e4-cde3-4258-9484-45bcd3a10cea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:17:33.165545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.165462 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.165545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.165490 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.165545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.165500 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn9z\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-kube-api-access-pmn9z\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.165545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.165510 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.165545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.165519 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.165545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.165527 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce384e4-cde3-4258-9484-45bcd3a10cea-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:33.482241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.482137 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7"] Nov 01 15:17:33.586889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.586851 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx"] Nov 01 15:17:33.740442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.740353 5473 generic.go:356] "Generic (PLEG): container finished" podID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerID="043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896" exitCode=0 Nov 01 15:17:33.740442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.740408 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" event={"ID":"cce384e4-cde3-4258-9484-45bcd3a10cea","Type":"ContainerDied","Data":"043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896"} Nov 01 15:17:33.740442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.740415 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" Nov 01 15:17:33.740770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.740448 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg" event={"ID":"cce384e4-cde3-4258-9484-45bcd3a10cea","Type":"ContainerDied","Data":"6ec5e99351a61b75b52359242c29990983d7826d9d9fdb0a37a3cd780d04e446"} Nov 01 15:17:33.740770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.740470 5473 scope.go:122] "RemoveContainer" containerID="043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896" Nov 01 15:17:33.751839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.751686 5473 scope.go:122] "RemoveContainer" containerID="73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498" Nov 01 15:17:33.762369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.762347 5473 scope.go:122] "RemoveContainer" containerID="043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896" Nov 01 15:17:33.762700 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:33.762680 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896\": container with ID starting with 043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896 not found: ID does not exist" containerID="043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896" Nov 01 15:17:33.762809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.762713 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896"} err="failed to get container status \"043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896\": rpc error: code = NotFound desc = could not find container \"043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896\": container with ID starting with 043297bdca6dd6b16221564619da44e56e041e81b97ff7ee95a5dabcbff2a896 not found: ID does not exist" Nov 01 15:17:33.762809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.762738 5473 scope.go:122] "RemoveContainer" containerID="73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498" Nov 01 15:17:33.763031 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:33.763014 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498\": container with ID starting with 73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498 not found: ID does not exist" containerID="73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498" Nov 01 15:17:33.763153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.763042 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498"} err="failed to get container status \"73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498\": rpc error: code = NotFound desc = could not find container \"73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498\": container with ID starting with 73f8312ad9cf8aac49548fe17a7f04302580d3a4bb6a106cc1e642d8a423e498 not found: ID does not exist" Nov 01 15:17:33.854874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.854844 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:33.907391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.907359 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg"] Nov 01 15:17:33.910630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:33.910604 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2cf1ee-openshift-default-847c6675bf-2fxrg"] Nov 01 15:17:34.121973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:34.121942 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:34.541688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:34.541656 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:35.095419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:35.095383 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" path="/var/lib/kubelet/pods/cce384e4-cde3-4258-9484-45bcd3a10cea/volumes" Nov 01 15:17:35.664094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:35.664055 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:35.817644 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:35.817610 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8"] Nov 01 15:17:35.923563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:35.923483 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5"] Nov 01 15:17:36.209527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:36.209448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:36.363638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:36.363600 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:36.522692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:36.522659 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:38.483554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.483509 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" podUID="4b8d1693-f28a-42c7-81c5-3577e2438f3a" containerName="payload-processing" containerID="cri-o://6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526" gracePeriod=25 Nov 01 15:17:38.587292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.587249 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" podUID="cb49e057-732e-4399-b9c6-8e238f50e9b9" containerName="payload-pre-processing" containerID="cri-o://0e76d79f1caa9f9a0a43fd01952f9a915748606aff7d639a7b7ed4c1edd90f30" gracePeriod=25 Nov 01 15:17:38.752103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.752037 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:17:38.782738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.782686 5473 generic.go:356] "Generic (PLEG): container finished" podID="cb49e057-732e-4399-b9c6-8e238f50e9b9" containerID="0e76d79f1caa9f9a0a43fd01952f9a915748606aff7d639a7b7ed4c1edd90f30" exitCode=0 Nov 01 15:17:38.782922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.782784 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" event={"ID":"cb49e057-732e-4399-b9c6-8e238f50e9b9","Type":"ContainerDied","Data":"0e76d79f1caa9f9a0a43fd01952f9a915748606aff7d639a7b7ed4c1edd90f30"} Nov 01 15:17:38.785429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.785405 5473 generic.go:356] "Generic (PLEG): container finished" podID="4b8d1693-f28a-42c7-81c5-3577e2438f3a" containerID="6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526" exitCode=0 Nov 01 15:17:38.785555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.785456 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" Nov 01 15:17:38.785555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.785494 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" event={"ID":"4b8d1693-f28a-42c7-81c5-3577e2438f3a","Type":"ContainerDied","Data":"6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526"} Nov 01 15:17:38.785689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.785561 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7" event={"ID":"4b8d1693-f28a-42c7-81c5-3577e2438f3a","Type":"ContainerDied","Data":"57dc8c996ba51e139eb970e0e512f599e0dfffad58412e138ac4b9e4a545c5f9"} Nov 01 15:17:38.785689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.785596 5473 scope.go:122] "RemoveContainer" containerID="6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526" Nov 01 15:17:38.796730 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.796707 5473 scope.go:122] "RemoveContainer" containerID="6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526" Nov 01 15:17:38.797017 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:38.796994 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526\": container with ID starting with 6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526 not found: ID does not exist" containerID="6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526" Nov 01 15:17:38.797069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.797026 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526"} err="failed to get container status \"6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526\": rpc error: code = NotFound desc = could not find container \"6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526\": container with ID starting with 6e53afeee41c90339876f29ae46f5de420a6d8eb79753f2a6bfc6fd4ccaad526 not found: ID does not exist" Nov 01 15:17:38.853376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.853352 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn\" (UniqueName: \"kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn\") pod \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " Nov 01 15:17:38.853489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.853449 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume\") pod \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\" (UID: \"4b8d1693-f28a-42c7-81c5-3577e2438f3a\") " Nov 01 15:17:38.853867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.853843 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume" pod "4b8d1693-f28a-42c7-81c5-3577e2438f3a" (UID: "4b8d1693-f28a-42c7-81c5-3577e2438f3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:17:38.854100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.854082 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8d1693-f28a-42c7-81c5-3577e2438f3a-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:38.855649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.855615 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn" pod "4b8d1693-f28a-42c7-81c5-3577e2438f3a" (UID: "4b8d1693-f28a-42c7-81c5-3577e2438f3a"). InnerVolumeSpecName "kube-api-access-zgqnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:38.865068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.865044 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:17:38.955563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.955527 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume\") pod \"cb49e057-732e-4399-b9c6-8e238f50e9b9\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " Nov 01 15:17:38.955743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.955575 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5\" (UniqueName: \"kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5\") pod \"cb49e057-732e-4399-b9c6-8e238f50e9b9\" (UID: \"cb49e057-732e-4399-b9c6-8e238f50e9b9\") " Nov 01 15:17:38.956000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.955967 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume" pod "cb49e057-732e-4399-b9c6-8e238f50e9b9" (UID: "cb49e057-732e-4399-b9c6-8e238f50e9b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:17:38.956173 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.956157 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb49e057-732e-4399-b9c6-8e238f50e9b9-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:38.956209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.956176 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqnn\" (UniqueName: \"kubernetes.io/projected/4b8d1693-f28a-42c7-81c5-3577e2438f3a-kube-api-access-zgqnn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:38.957702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:38.957679 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5" pod "cb49e057-732e-4399-b9c6-8e238f50e9b9" (UID: "cb49e057-732e-4399-b9c6-8e238f50e9b9"). InnerVolumeSpecName "kube-api-access-7bjj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:39.057925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.057889 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjj5\" (UniqueName: \"kubernetes.io/projected/cb49e057-732e-4399-b9c6-8e238f50e9b9-kube-api-access-7bjj5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:39.211511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.211474 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7"] Nov 01 15:17:39.216866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.216838 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e92165-7c4d64757-gl2l7"] Nov 01 15:17:39.795581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.795551 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" Nov 01 15:17:39.796099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.795550 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx" event={"ID":"cb49e057-732e-4399-b9c6-8e238f50e9b9","Type":"ContainerDied","Data":"e3a49af79d12e46ca5a90fee32214152ee81c796123b05c1f25205f740349149"} Nov 01 15:17:39.796099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.795683 5473 scope.go:122] "RemoveContainer" containerID="0e76d79f1caa9f9a0a43fd01952f9a915748606aff7d639a7b7ed4c1edd90f30" Nov 01 15:17:39.960872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.960841 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx"] Nov 01 15:17:39.964868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:39.964838 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e92165-664c984df9-wpjcx"] Nov 01 15:17:40.154369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:40.154280 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:40.818284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:40.818244 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" podUID="8e4bbb61-4734-4502-b0d0-27f3b2ef4958" containerName="payload-processing" containerID="cri-o://50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730" gracePeriod=25 Nov 01 15:17:40.924299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:40.924257 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" podUID="253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" containerName="payload-pre-processing" containerID="cri-o://b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d" gracePeriod=25 Nov 01 15:17:41.086362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.086245 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8d1693-f28a-42c7-81c5-3577e2438f3a" path="/var/lib/kubelet/pods/4b8d1693-f28a-42c7-81c5-3577e2438f3a/volumes" Nov 01 15:17:41.086861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.086807 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb49e057-732e-4399-b9c6-8e238f50e9b9" path="/var/lib/kubelet/pods/cb49e057-732e-4399-b9c6-8e238f50e9b9/volumes" Nov 01 15:17:41.089674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.089653 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:17:41.126192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.126162 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:41.193404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.193368 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume\") pod \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " Nov 01 15:17:41.193707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.193443 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c\" (UniqueName: \"kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c\") pod \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\" (UID: \"8e4bbb61-4734-4502-b0d0-27f3b2ef4958\") " Nov 01 15:17:41.193966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.193935 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume" pod "8e4bbb61-4734-4502-b0d0-27f3b2ef4958" (UID: "8e4bbb61-4734-4502-b0d0-27f3b2ef4958"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:17:41.194171 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.194152 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:41.196437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.196405 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c" pod "8e4bbb61-4734-4502-b0d0-27f3b2ef4958" (UID: "8e4bbb61-4734-4502-b0d0-27f3b2ef4958"). InnerVolumeSpecName "kube-api-access-sz64c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:41.222313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.222281 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:17:41.286463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.286431 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:41.295209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.295180 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume\") pod \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " Nov 01 15:17:41.295478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.295316 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv\" (UniqueName: \"kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv\") pod \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\" (UID: \"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c\") " Nov 01 15:17:41.295643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.295615 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume" pod "253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" (UID: "253446fe-1c2b-4edd-a0d1-08b2a0a36b7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:17:41.295849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.295831 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:41.295912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.295853 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz64c\" (UniqueName: \"kubernetes.io/projected/8e4bbb61-4734-4502-b0d0-27f3b2ef4958-kube-api-access-sz64c\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:41.297474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.297449 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv" pod "253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" (UID: "253446fe-1c2b-4edd-a0d1-08b2a0a36b7c"). InnerVolumeSpecName "kube-api-access-rkttv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:17:41.396947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.396858 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkttv\" (UniqueName: \"kubernetes.io/projected/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c-kube-api-access-rkttv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:17:41.814320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.814282 5473 generic.go:356] "Generic (PLEG): container finished" podID="8e4bbb61-4734-4502-b0d0-27f3b2ef4958" containerID="50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730" exitCode=0 Nov 01 15:17:41.814549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.814359 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" Nov 01 15:17:41.814549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.814376 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" event={"ID":"8e4bbb61-4734-4502-b0d0-27f3b2ef4958","Type":"ContainerDied","Data":"50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730"} Nov 01 15:17:41.814549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.814473 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8" event={"ID":"8e4bbb61-4734-4502-b0d0-27f3b2ef4958","Type":"ContainerDied","Data":"a4cbc7b6ee623409feb0c397846d9e2328138bda0a35811fde101dee16a3d640"} Nov 01 15:17:41.814549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.814504 5473 scope.go:122] "RemoveContainer" containerID="50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730" Nov 01 15:17:41.816225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.816190 5473 generic.go:356] "Generic (PLEG): container finished" podID="253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" containerID="b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d" exitCode=0 Nov 01 15:17:41.816472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.816454 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" event={"ID":"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c","Type":"ContainerDied","Data":"b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d"} Nov 01 15:17:41.816555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.816501 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" event={"ID":"253446fe-1c2b-4edd-a0d1-08b2a0a36b7c","Type":"ContainerDied","Data":"1c08f83744b1306d73a0b9971a53ec901cae094ab84b3bad2797efaac7122de7"} Nov 01 15:17:41.816628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.816571 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5" Nov 01 15:17:41.826943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.826673 5473 scope.go:122] "RemoveContainer" containerID="50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730" Nov 01 15:17:41.827172 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:41.826981 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730\": container with ID starting with 50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730 not found: ID does not exist" containerID="50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730" Nov 01 15:17:41.827172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.827008 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730"} err="failed to get container status \"50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730\": rpc error: code = NotFound desc = could not find container \"50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730\": container with ID starting with 50349a9b19f1b7317c1e3b516aaced354a19dda83fab60e32475b31699362730 not found: ID does not exist" Nov 01 15:17:41.827172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.827026 5473 scope.go:122] "RemoveContainer" containerID="b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d" Nov 01 15:17:41.844241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.844191 5473 scope.go:122] "RemoveContainer" containerID="b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d" Nov 01 15:17:41.844546 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:17:41.844526 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d\": container with ID starting with b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d not found: ID does not exist" containerID="b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d" Nov 01 15:17:41.844604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.844554 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d"} err="failed to get container status \"b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d\": rpc error: code = NotFound desc = could not find container \"b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d\": container with ID starting with b30084b04bc7e4fddc9953f960682e2dd8777d2cb36b7fffe86758e98ad6aa2d not found: ID does not exist" Nov 01 15:17:41.980443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.980407 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8"] Nov 01 15:17:41.985565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:41.985541 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-926972-594bf48fc-nrlw8"] Nov 01 15:17:42.098938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:42.098838 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5"] Nov 01 15:17:42.102002 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:42.101975 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-926972-5689bfcbbc-jqrq5"] Nov 01 15:17:43.082918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:43.082887 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" path="/var/lib/kubelet/pods/253446fe-1c2b-4edd-a0d1-08b2a0a36b7c/volumes" Nov 01 15:17:43.083401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:43.083385 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4bbb61-4734-4502-b0d0-27f3b2ef4958" path="/var/lib/kubelet/pods/8e4bbb61-4734-4502-b0d0-27f3b2ef4958/volumes" Nov 01 15:17:44.153465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:44.153439 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:17:44.159360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:44.159330 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:17:45.306972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:45.306933 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:47.135324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:47.131640 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:47.403599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:47.403466 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:47.590224 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:47.590175 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:48.915315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:48.915276 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:49.455184 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:49.455136 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:50.226496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:50.226451 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:51.818068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:51.818030 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:51.994910 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:51.994876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:52.156982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:52.156892 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:56.007039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:56.007000 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf"] Nov 01 15:17:56.354404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:56.354369 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:56.529267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:56.529237 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:56.733987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:56.733881 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:17:56.940395 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:17:56.940350 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:01.007579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.007541 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" podUID="2083f500-32e8-40b7-91d0-dfc6f1ddea86" containerName="payload-pre-processing" containerID="cri-o://246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe" gracePeriod=25 Nov 01 15:18:01.114892 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:18:01.114849 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:18:01.194495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.194460 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:01.318442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.318415 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:18:01.460355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.459511 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf\" (UniqueName: \"kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf\") pod \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " Nov 01 15:18:01.460355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.459694 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume\") pod \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\" (UID: \"2083f500-32e8-40b7-91d0-dfc6f1ddea86\") " Nov 01 15:18:01.460862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.460832 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume" pod "2083f500-32e8-40b7-91d0-dfc6f1ddea86" (UID: "2083f500-32e8-40b7-91d0-dfc6f1ddea86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:18:01.463408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.463378 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf" pod "2083f500-32e8-40b7-91d0-dfc6f1ddea86" (UID: "2083f500-32e8-40b7-91d0-dfc6f1ddea86"). InnerVolumeSpecName "kube-api-access-fnqqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:18:01.560997 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.560912 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnqqf\" (UniqueName: \"kubernetes.io/projected/2083f500-32e8-40b7-91d0-dfc6f1ddea86-kube-api-access-fnqqf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:18:01.560997 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:01.560945 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2083f500-32e8-40b7-91d0-dfc6f1ddea86-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:18:02.018969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.018932 5473 generic.go:356] "Generic (PLEG): container finished" podID="2083f500-32e8-40b7-91d0-dfc6f1ddea86" containerID="246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe" exitCode=0 Nov 01 15:18:02.019691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.019660 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" Nov 01 15:18:02.019835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.019610 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" event={"ID":"2083f500-32e8-40b7-91d0-dfc6f1ddea86","Type":"ContainerDied","Data":"246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe"} Nov 01 15:18:02.019930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.019831 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf" event={"ID":"2083f500-32e8-40b7-91d0-dfc6f1ddea86","Type":"ContainerDied","Data":"5330882a8fc1bab6bcbdf495309d43ec84a1a1c6a91b5a787f62800b5dde097c"} Nov 01 15:18:02.019930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.019883 5473 scope.go:122] "RemoveContainer" containerID="246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe" Nov 01 15:18:02.052564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.052536 5473 scope.go:122] "RemoveContainer" containerID="246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe" Nov 01 15:18:02.052937 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:18:02.052909 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe\": container with ID starting with 246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe not found: ID does not exist" containerID="246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe" Nov 01 15:18:02.053010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.052949 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe"} err="failed to get container status \"246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe\": rpc error: code = NotFound desc = could not find container \"246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe\": container with ID starting with 246c9c981def35f47e657c97bad979ca496f557463b7229b90959366d81facfe not found: ID does not exist" Nov 01 15:18:02.185331 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.185293 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf"] Nov 01 15:18:02.235232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.235183 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-c89271-5cb646dcb7-vqzqf"] Nov 01 15:18:02.235964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.235937 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:02.395919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.395827 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:02.876020 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:02.875981 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:03.083343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:03.083310 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2083f500-32e8-40b7-91d0-dfc6f1ddea86" path="/var/lib/kubelet/pods/2083f500-32e8-40b7-91d0-dfc6f1ddea86/volumes" Nov 01 15:18:07.272480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:07.272449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:08.378651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:08.378615 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:08.567081 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:08.567047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:09.884881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:09.884843 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:10.491679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:10.491641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:12.048435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:12.048398 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:12.449603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:12.449499 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:12.691517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:12.691484 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:16.310948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:16.310902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:16.993879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:16.993841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:19.123579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.123541 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 01 15:18:19.124097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.123607 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 01 15:18:19.156151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.156117 5473 generic.go:356] "Generic (PLEG): container finished" podID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerID="3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006" exitCode=0 Nov 01 15:18:19.156339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.156190 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" event={"ID":"6ef29d51-4057-4f6c-8533-c03c355a53db","Type":"ContainerDied","Data":"3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006"} Nov 01 15:18:19.156653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.156637 5473 scope.go:122] "RemoveContainer" containerID="3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006" Nov 01 15:18:19.158175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.158146 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:19.158269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.158230 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:19.158269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:19.158263 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:20.165332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:20.165302 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" event={"ID":"6ef29d51-4057-4f6c-8533-c03c355a53db","Type":"ContainerStarted","Data":"dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a"} Nov 01 15:18:21.078929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:21.078897 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 01 15:18:21.079138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:21.078945 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 01 15:18:21.083067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:21.083040 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:18:21.404841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:21.404756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:22.075789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:22.075758 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 01 15:18:22.075972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:22.075821 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 01 15:18:23.075897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:23.075860 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 01 15:18:23.076349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:23.075923 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 01 15:18:24.076263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:24.076194 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 01 15:18:24.076692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:24.076291 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 01 15:18:25.075590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:25.075553 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 01 15:18:25.075780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:25.075618 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 01 15:18:25.770941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:25.770906 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:26.012426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:26.012382 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:26.080688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:26.080613 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:18:26.081084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:26.081065 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:18:26.081753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:26.081736 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:18:30.206165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:30.206124 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:31.108705 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:18:31.108667 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:18:34.700904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:34.700861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:39.155451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:39.155416 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:43.691554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:43.691518 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:47.598559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.598526 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:47.679858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.679827 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j"] Nov 01 15:18:47.680502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680486 5473 cpu_manager.go:414] "containerMap: removing container" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680503 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680523 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4b8d1693-f28a-42c7-81c5-3577e2438f3a" containerName="payload-processing" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680529 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8d1693-f28a-42c7-81c5-3577e2438f3a" containerName="payload-processing" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680543 5473 cpu_manager.go:414] "containerMap: removing container" podUID="cb49e057-732e-4399-b9c6-8e238f50e9b9" containerName="payload-pre-processing" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680548 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb49e057-732e-4399-b9c6-8e238f50e9b9" containerName="payload-pre-processing" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680559 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8e4bbb61-4734-4502-b0d0-27f3b2ef4958" containerName="payload-processing" Nov 01 15:18:47.680571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680564 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4bbb61-4734-4502-b0d0-27f3b2ef4958" containerName="payload-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680576 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2083f500-32e8-40b7-91d0-dfc6f1ddea86" containerName="payload-pre-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680580 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2083f500-32e8-40b7-91d0-dfc6f1ddea86" containerName="payload-pre-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680592 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680596 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680604 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680608 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680619 5473 cpu_manager.go:414] "containerMap: removing container" podUID="253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" containerName="payload-pre-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680624 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" containerName="payload-pre-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680630 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0d3fe657-30d8-45b6-82f0-91b3532227ea" containerName="main" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680634 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3fe657-30d8-45b6-82f0-91b3532227ea" containerName="main" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680714 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680722 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b49dc5d-d83c-4bf9-9f57-ef5d6cae298a" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680730 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="253446fe-1c2b-4edd-a0d1-08b2a0a36b7c" containerName="payload-pre-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680734 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3fe657-30d8-45b6-82f0-91b3532227ea" containerName="main" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680744 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680752 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8d1693-f28a-42c7-81c5-3577e2438f3a" containerName="payload-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680759 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680765 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb49e057-732e-4399-b9c6-8e238f50e9b9" containerName="payload-pre-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680772 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4bbb61-4734-4502-b0d0-27f3b2ef4958" containerName="payload-processing" Nov 01 15:18:47.680807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.680778 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2083f500-32e8-40b7-91d0-dfc6f1ddea86" containerName="payload-pre-processing" Nov 01 15:18:47.688476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.688446 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.739379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.739342 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j"] Nov 01 15:18:47.785018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.784990 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:18:47.785194 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.785092 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:18:47.844307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.844275 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.844493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.844320 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.844493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.844461 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc46d\" (UniqueName: \"kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.945511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.945415 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.945511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.945477 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.945772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.945577 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc46d\" (UniqueName: \"kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.946427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.946398 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:47.947932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:47.947911 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:48.002917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:48.002888 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc46d\" (UniqueName: \"kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d\") pod \"maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:48.302683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:48.302645 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:48.672488 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:18:48.672450 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43915c8e_d254_4f89_88b3_715566829754.slice/crio-98e5b54c2fd14246f0dfc8315618ee4705a5a09ce740392a562a63920f87a4b6 WatchSource:0}: Error finding container 98e5b54c2fd14246f0dfc8315618ee4705a5a09ce740392a562a63920f87a4b6: Status 404 returned error can't find the container with id 98e5b54c2fd14246f0dfc8315618ee4705a5a09ce740392a562a63920f87a4b6 Nov 01 15:18:48.676558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:48.676533 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j"] Nov 01 15:18:49.253761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:49.253669 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:49.380574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:49.380536 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" event={"ID":"43915c8e-d254-4f89-88b3-715566829754","Type":"ContainerStarted","Data":"ef7a53db44365610da6cf7cca0f8c3403fd5798b0ff1448c79a0f2d48128c2bc"} Nov 01 15:18:49.380782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:49.380622 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" event={"ID":"43915c8e-d254-4f89-88b3-715566829754","Type":"ContainerStarted","Data":"98e5b54c2fd14246f0dfc8315618ee4705a5a09ce740392a562a63920f87a4b6"} Nov 01 15:18:49.497390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:49.497335 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" podStartSLOduration=2.49732023 podStartE2EDuration="2.49732023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:18:49.493856048 +0000 UTC m=+3668.552007898" watchObservedRunningTime="2026-11-01 15:18:49.49732023 +0000 UTC m=+3668.555472080" Nov 01 15:18:50.390299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.390262 5473 generic.go:356] "Generic (PLEG): container finished" podID="43915c8e-d254-4f89-88b3-715566829754" containerID="ef7a53db44365610da6cf7cca0f8c3403fd5798b0ff1448c79a0f2d48128c2bc" exitCode=0 Nov 01 15:18:50.390720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.390337 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" event={"ID":"43915c8e-d254-4f89-88b3-715566829754","Type":"ContainerDied","Data":"ef7a53db44365610da6cf7cca0f8c3403fd5798b0ff1448c79a0f2d48128c2bc"} Nov 01 15:18:50.390720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.390477 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n"] Nov 01 15:18:50.391079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.391066 5473 cpu_manager.go:414] "containerMap: removing container" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" Nov 01 15:18:50.391117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.391081 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce384e4-cde3-4258-9484-45bcd3a10cea" containerName="istio-proxy" Nov 01 15:18:50.394567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.394549 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.446827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.446795 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw"] Nov 01 15:18:50.452785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.452754 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.474435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.474399 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n"] Nov 01 15:18:50.482559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482520 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.482559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482560 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.482808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482662 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.482808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482747 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.482808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482793 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.482996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482917 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.482996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.482951 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.483132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.483015 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.483132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.483079 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rb6\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.486791 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.486758 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58vzm\"" Nov 01 15:18:50.499687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.499652 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw"] Nov 01 15:18:50.584395 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584361 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.584570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584412 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.584570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584483 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.584570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584545 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.584749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584597 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczjx\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.584749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584622 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rb6\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.584864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584759 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.584864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584793 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.584864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584831 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.585040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584896 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584927 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.585040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.584941 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585016 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585055 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585158 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585207 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585269 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.585332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585233 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585315 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.585332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585318 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585354 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.585575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585553 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.585824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.585800 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.587338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.587314 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.587566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.587547 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.642110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.642037 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.642309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.642253 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rb6\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6\") pod \"maas-default-gateway-openshift-default-7f5667f86c-h9b2n\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.686674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.686608 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.686888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.686774 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczjx\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.686978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.686911 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.686978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.686944 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.686980 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687042 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687124 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687171 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687202 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687439 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687507 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687543 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687652 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.687753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.687730 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.689345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.689323 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.689613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.689595 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.708719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.708692 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:50.752961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.752931 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.753488 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.753458 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczjx\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-stknw\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:50.775702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:50.775672 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:51.133183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.133149 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n"] Nov 01 15:18:51.164433 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:18:51.164402 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc919f6f7_a12e_4fa4_a6d8_8b0adc060037.slice/crio-dd56e3fb7fb339cfbe083bce40214de44ac2d51ede8cf6bae5d2d367d016e7c2 WatchSource:0}: Error finding container dd56e3fb7fb339cfbe083bce40214de44ac2d51ede8cf6bae5d2d367d016e7c2: Status 404 returned error can't find the container with id dd56e3fb7fb339cfbe083bce40214de44ac2d51ede8cf6bae5d2d367d016e7c2 Nov 01 15:18:51.167075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.167042 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:51.167162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.167120 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:51.167162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.167150 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:51.178368 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:18:51.178344 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4c58e2_dded_4c0a_82d7_23240410d460.slice/crio-cf6ce9c029d680a66646386b4fc19a3b7457098e9f3b68e90979fee39599f4c6 WatchSource:0}: Error finding container cf6ce9c029d680a66646386b4fc19a3b7457098e9f3b68e90979fee39599f4c6: Status 404 returned error can't find the container with id cf6ce9c029d680a66646386b4fc19a3b7457098e9f3b68e90979fee39599f4c6 Nov 01 15:18:51.181777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.181206 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:51.181777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.181309 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:51.181777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.181353 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:18:51.182955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.182933 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw"] Nov 01 15:18:51.399339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.399211 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" event={"ID":"ea4c58e2-dded-4c0a-82d7-23240410d460","Type":"ContainerStarted","Data":"3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a"} Nov 01 15:18:51.399339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.399312 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" event={"ID":"ea4c58e2-dded-4c0a-82d7-23240410d460","Type":"ContainerStarted","Data":"cf6ce9c029d680a66646386b4fc19a3b7457098e9f3b68e90979fee39599f4c6"} Nov 01 15:18:51.401274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.401243 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" event={"ID":"c919f6f7-a12e-4fa4-a6d8-8b0adc060037","Type":"ContainerStarted","Data":"90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1"} Nov 01 15:18:51.401409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.401292 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" event={"ID":"c919f6f7-a12e-4fa4-a6d8-8b0adc060037","Type":"ContainerStarted","Data":"dd56e3fb7fb339cfbe083bce40214de44ac2d51ede8cf6bae5d2d367d016e7c2"} Nov 01 15:18:51.524264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.524178 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podStartSLOduration=1.5241558880000001 podStartE2EDuration="1.524155888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:18:51.516415729 +0000 UTC m=+3670.574567583" watchObservedRunningTime="2026-11-01 15:18:51.524155888 +0000 UTC m=+3670.582307739" Nov 01 15:18:51.570908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.570883 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:51.642926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.642857 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podStartSLOduration=1.642838901 podStartE2EDuration="1.642838901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:18:51.640408971 +0000 UTC m=+3670.698560821" watchObservedRunningTime="2026-11-01 15:18:51.642838901 +0000 UTC m=+3670.700990753" Nov 01 15:18:51.701245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.701085 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d\" (UniqueName: \"kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d\") pod \"43915c8e-d254-4f89-88b3-715566829754\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " Nov 01 15:18:51.701245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.701183 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca\") pod \"43915c8e-d254-4f89-88b3-715566829754\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " Nov 01 15:18:51.701576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.701373 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca\") pod \"43915c8e-d254-4f89-88b3-715566829754\" (UID: \"43915c8e-d254-4f89-88b3-715566829754\") " Nov 01 15:18:51.701809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.701784 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca" pod "43915c8e-d254-4f89-88b3-715566829754" (UID: "43915c8e-d254-4f89-88b3-715566829754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:18:51.701941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.701925 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43915c8e-d254-4f89-88b3-715566829754-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:18:51.703402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.703375 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d" pod "43915c8e-d254-4f89-88b3-715566829754" (UID: "43915c8e-d254-4f89-88b3-715566829754"). InnerVolumeSpecName "kube-api-access-hc46d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:18:51.703514 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.703449 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca" pod "43915c8e-d254-4f89-88b3-715566829754" (UID: "43915c8e-d254-4f89-88b3-715566829754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:18:51.709718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.709695 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:18:51.776143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.776104 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:18:51.803096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.803054 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc46d\" (UniqueName: \"kubernetes.io/projected/43915c8e-d254-4f89-88b3-715566829754-kube-api-access-hc46d\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:18:51.803096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:51.803098 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43915c8e-d254-4f89-88b3-715566829754-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:18:52.409754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.409719 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" Nov 01 15:18:52.409754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.409736 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j" event={"ID":"43915c8e-d254-4f89-88b3-715566829754","Type":"ContainerDied","Data":"98e5b54c2fd14246f0dfc8315618ee4705a5a09ce740392a562a63920f87a4b6"} Nov 01 15:18:52.410363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.409828 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98e5b54c2fd14246f0dfc8315618ee4705a5a09ce740392a562a63920f87a4b6" Nov 01 15:18:52.710490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.710407 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:52.710490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.710471 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:52.779285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.779243 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:52.779478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:52.779314 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:53.358596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:53.355752 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:53.709922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:53.709835 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:53.709922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:53.709891 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:53.776091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:53.776050 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:53.776316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:53.776125 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:54.709382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:54.709343 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:54.709564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:54.709405 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:54.776320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:54.776284 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:54.776736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:54.776338 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:54.957238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:54.942206 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:55.291282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:55.291237 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:55.513771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:55.513733 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:55.709538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:55.709451 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:55.709538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:55.709506 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:55.776888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:55.776852 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:55.777410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:55.777376 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:56.709759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:56.709726 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:56.709952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:56.709782 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:56.778118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:56.776831 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:56.778118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:56.776894 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:56.782633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:56.782602 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:57.276595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.276348 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:57.642302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.642167 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:57.709141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.709091 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:57.709382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.709158 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:57.776633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.776588 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:57.776860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.776647 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:57.803445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:57.803410 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:58.710165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:58.710120 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" start-of-body= Nov 01 15:18:58.710389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:58.710195 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": dial tcp 10.134.1.33:15021: connect: connection refused" Nov 01 15:18:58.764964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:58.764928 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:58.776185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:58.776153 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" start-of-body= Nov 01 15:18:58.776339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:58.776204 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": dial tcp 10.134.1.34:15021: connect: connection refused" Nov 01 15:18:59.006307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:59.003074 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:59.456798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:59.456758 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:59.855684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:59.855649 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:18:59.906860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:59.906811 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj"] Nov 01 15:18:59.907418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:18:59.907366 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" containerID="cri-o://f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72" gracePeriod=30 Nov 01 15:19:00.041141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.039779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:00.223692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.223602 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:19:00.224076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.224054 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:19:00.225005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.224987 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:19:00.249884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.249849 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:19:00.250281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.250252 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:19:00.251201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.251173 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:19:00.636538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:00.636501 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:01.108928 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:19:01.108891 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:19:02.569446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:02.569384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:04.735933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:04.735859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:04.802729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:04.802696 5473 patch_prober.go:29] interesting pod/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:19:04.802911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:04.802760 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:19:05.194408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.194383 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:19:05.332372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332272 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332322 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332350 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332402 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332581 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332632 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332672 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332724 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.332849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332759 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m\") pod \"6abf39f2-78ae-44ce-801b-dcaab11f5802\" (UID: \"6abf39f2-78ae-44ce-801b-dcaab11f5802\") " Nov 01 15:19:05.333025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332855 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:19:05.333025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.332929 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:19:05.333168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333051 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:19:05.333168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333064 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:19:05.333297 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333245 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:19:05.333503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333473 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.333615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333510 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.333615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333526 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.333615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333541 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf39f2-78ae-44ce-801b-dcaab11f5802-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.333615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.333556 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.334925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.334897 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:19:05.335063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.335043 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:19:05.335350 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.335325 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:19:05.335433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.335334 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m" pod "6abf39f2-78ae-44ce-801b-dcaab11f5802" (UID: "6abf39f2-78ae-44ce-801b-dcaab11f5802"). InnerVolumeSpecName "kube-api-access-jhb4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:19:05.435114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.435069 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhb4m\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-kube-api-access-jhb4m\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.435114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.435106 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.435114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.435117 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.435417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.435127 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf39f2-78ae-44ce-801b-dcaab11f5802-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:19:05.564670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.564633 5473 generic.go:356] "Generic (PLEG): container finished" podID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerID="f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72" exitCode=0 Nov 01 15:19:05.564858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.564690 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" Nov 01 15:19:05.564858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.564715 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" event={"ID":"6abf39f2-78ae-44ce-801b-dcaab11f5802","Type":"ContainerDied","Data":"f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72"} Nov 01 15:19:05.564858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.564778 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj" event={"ID":"6abf39f2-78ae-44ce-801b-dcaab11f5802","Type":"ContainerDied","Data":"475c5129b713d87e74ba9b6843efcc0c53393f12d59ebbd8e34682708bf35427"} Nov 01 15:19:05.564858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.564810 5473 scope.go:122] "RemoveContainer" containerID="f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72" Nov 01 15:19:05.576746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.576725 5473 scope.go:122] "RemoveContainer" containerID="a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514" Nov 01 15:19:05.587429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.587406 5473 scope.go:122] "RemoveContainer" containerID="f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72" Nov 01 15:19:05.587728 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:19:05.587705 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72\": container with ID starting with f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72 not found: ID does not exist" containerID="f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72" Nov 01 15:19:05.587786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.587738 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72"} err="failed to get container status \"f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72\": rpc error: code = NotFound desc = could not find container \"f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72\": container with ID starting with f596e882683d4a6021aa01f7acfddd37655fc4a20869dc777bd6e1b1aa193d72 not found: ID does not exist" Nov 01 15:19:05.587786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.587757 5473 scope.go:122] "RemoveContainer" containerID="a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514" Nov 01 15:19:05.587990 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:19:05.587977 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514\": container with ID starting with a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514 not found: ID does not exist" containerID="a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514" Nov 01 15:19:05.588043 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.587994 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514"} err="failed to get container status \"a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514\": rpc error: code = NotFound desc = could not find container \"a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514\": container with ID starting with a957b9d72a223b9924809efbb453dd781404bfd79cbec3cd4874534d23155514 not found: ID does not exist" Nov 01 15:19:05.731169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.731129 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj"] Nov 01 15:19:05.734388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:05.734361 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-926972-openshift-default-5ff7698fbf-hqsrj"] Nov 01 15:19:06.570926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:06.570873 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:07.082987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:07.082950 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" path="/var/lib/kubelet/pods/6abf39f2-78ae-44ce-801b-dcaab11f5802/volumes" Nov 01 15:19:07.441384 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:07.441297 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:07.603638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:07.603600 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:09.581969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:09.581931 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:12.141795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:12.141759 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:14.285598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:14.285549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:18.910954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:18.910923 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:19.785611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:19.785578 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:20.650165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650086 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw"] Nov 01 15:19:20.650773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650756 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" Nov 01 15:19:20.650821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650775 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" Nov 01 15:19:20.650821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650785 5473 cpu_manager.go:414] "containerMap: removing container" podUID="43915c8e-d254-4f89-88b3-715566829754" containerName="revoke-keys" Nov 01 15:19:20.650821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650789 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43915c8e-d254-4f89-88b3-715566829754" containerName="revoke-keys" Nov 01 15:19:20.650821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650815 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" Nov 01 15:19:20.650821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650821 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" Nov 01 15:19:20.650989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650937 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" Nov 01 15:19:20.650989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650946 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="43915c8e-d254-4f89-88b3-715566829754" containerName="revoke-keys" Nov 01 15:19:20.650989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.650955 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abf39f2-78ae-44ce-801b-dcaab11f5802" containerName="istio-proxy" Nov 01 15:19:20.654376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.654357 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762336 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762471 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762531 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762585 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762682 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762788 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762822 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.762925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.762903 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkkz\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.849947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.849910 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw"] Nov 01 15:19:20.864962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.864815 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.864962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.864868 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.864962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.864915 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkkz\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865001 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865075 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865160 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865203 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865268 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865315 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865334 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865716 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.865912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.865889 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.866148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.866128 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.867500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.866772 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.868822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.868798 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.870010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.869985 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.922925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.922848 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkkz\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.923319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.923296 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-2zlpw\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:20.960942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.960869 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:20.967985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:20.967959 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:21.392242 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:19:21.392189 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5c87e3_80a0_4c1b_827a_a7d78da38a99.slice/crio-e79f3a9b4c2167e50060f504cb8a44fa94fa11bd9515742447c77b812c217200 WatchSource:0}: Error finding container e79f3a9b4c2167e50060f504cb8a44fa94fa11bd9515742447c77b812c217200: Status 404 returned error can't find the container with id e79f3a9b4c2167e50060f504cb8a44fa94fa11bd9515742447c77b812c217200 Nov 01 15:19:21.396293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.396267 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw"] Nov 01 15:19:21.397577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.397544 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:19:21.397690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.397623 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:19:21.397690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.397669 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:19:21.709319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.708401 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" event={"ID":"fb5c87e3-80a0-4c1b-827a-a7d78da38a99","Type":"ContainerStarted","Data":"3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39"} Nov 01 15:19:21.709319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.708470 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" event={"ID":"fb5c87e3-80a0-4c1b-827a-a7d78da38a99","Type":"ContainerStarted","Data":"e79f3a9b4c2167e50060f504cb8a44fa94fa11bd9515742447c77b812c217200"} Nov 01 15:19:21.817445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.817414 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:21.828478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.828413 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podStartSLOduration=1.828391208 podStartE2EDuration="1.828391208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:21.826601566 +0000 UTC m=+3700.884753415" watchObservedRunningTime="2026-11-01 15:19:21.828391208 +0000 UTC m=+3700.886543059" Nov 01 15:19:21.968993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:21.968886 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:22.020315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:22.020277 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:22.970232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:22.970191 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:22.970643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:22.970278 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:23.704999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:23.704934 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:23.969484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:23.969394 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:23.969484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:23.969452 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:24.968522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:24.968490 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:24.968916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:24.968544 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:25.969117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:25.969081 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:25.969589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:25.969146 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:26.508944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:26.508910 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:26.695772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:26.695738 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:26.881974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:26.881938 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:26.969147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:26.969112 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:26.969577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:26.969164 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:27.969277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:27.969207 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:27.969684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:27.969298 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:28.735337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:28.735304 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:28.969083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:28.969046 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Nov 01 15:19:28.969308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:28.969114 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Nov 01 15:19:29.759277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:29.759205 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:29.943071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:29.943031 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:30.147552 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.147518 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7"] Nov 01 15:19:30.154150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.154127 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.198345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.198309 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:30.198751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.198718 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:30.199682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.199659 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:19:30.203434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.203409 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7"] Nov 01 15:19:30.234757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.234724 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-af2dd2\"" Nov 01 15:19:30.243608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.243555 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.244112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.244085 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2knh\" (UniqueName: \"kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh\") pod \"payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.345681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.345640 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.346033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.346016 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2knh\" (UniqueName: \"kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh\") pod \"payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.347345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.347317 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.362263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.361899 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk"] Nov 01 15:19:30.367868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.367840 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.382308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.382277 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk"] Nov 01 15:19:30.416850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.415955 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2knh\" (UniqueName: \"kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh\") pod \"payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.422735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.422705 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-af2dd2-dockercfg-7nzcv\"" Nov 01 15:19:30.447639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.447603 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.448713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.448681 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ldn\" (UniqueName: \"kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn\") pod \"payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.470012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.469967 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:30.559299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.558373 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.559299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.558520 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ldn\" (UniqueName: \"kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn\") pod \"payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.559545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.559339 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.616557 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.616519 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ldn\" (UniqueName: \"kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn\") pod \"payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.697445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.697360 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:30.853449 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:19:30.853419 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8285dada_373e_45fc_8a1b_1a9d1530a98e.slice/crio-acd5c76ce0d27f7bcd13fbb8ed5d81e60b86bc24771d1bb31ad7832d1970b32a WatchSource:0}: Error finding container acd5c76ce0d27f7bcd13fbb8ed5d81e60b86bc24771d1bb31ad7832d1970b32a: Status 404 returned error can't find the container with id acd5c76ce0d27f7bcd13fbb8ed5d81e60b86bc24771d1bb31ad7832d1970b32a Nov 01 15:19:30.857548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:30.857523 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7"] Nov 01 15:19:31.070647 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:19:31.070617 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164930f3_c671_4958_9498_58f04f270361.slice/crio-7de82e051e6f3c03180ba46660b2830c673e78f11ae21820fdddf2556b8071c1 WatchSource:0}: Error finding container 7de82e051e6f3c03180ba46660b2830c673e78f11ae21820fdddf2556b8071c1: Status 404 returned error can't find the container with id 7de82e051e6f3c03180ba46660b2830c673e78f11ae21820fdddf2556b8071c1 Nov 01 15:19:31.082681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.082619 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk"] Nov 01 15:19:31.110320 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:19:31.110288 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:19:31.813735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.813696 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" event={"ID":"164930f3-c671-4958-9498-58f04f270361","Type":"ContainerStarted","Data":"a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4"} Nov 01 15:19:31.814240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.813764 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:31.814240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.813788 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" event={"ID":"164930f3-c671-4958-9498-58f04f270361","Type":"ContainerStarted","Data":"7de82e051e6f3c03180ba46660b2830c673e78f11ae21820fdddf2556b8071c1"} Nov 01 15:19:31.815608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.815579 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" event={"ID":"8285dada-373e-45fc-8a1b-1a9d1530a98e","Type":"ContainerStarted","Data":"a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967"} Nov 01 15:19:31.815762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.815643 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:31.815762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.815670 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" event={"ID":"8285dada-373e-45fc-8a1b-1a9d1530a98e","Type":"ContainerStarted","Data":"acd5c76ce0d27f7bcd13fbb8ed5d81e60b86bc24771d1bb31ad7832d1970b32a"} Nov 01 15:19:31.934571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:31.934514 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" podStartSLOduration=1.934493692 podStartE2EDuration="1.934493692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:31.931152306 +0000 UTC m=+3710.989304155" watchObservedRunningTime="2026-11-01 15:19:31.934493692 +0000 UTC m=+3710.992645545" Nov 01 15:19:32.048299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:32.046366 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" podStartSLOduration=2.046345029 podStartE2EDuration="2.046345029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:19:32.045177351 +0000 UTC m=+3711.103329216" watchObservedRunningTime="2026-11-01 15:19:32.046345029 +0000 UTC m=+3711.104496882" Nov 01 15:19:33.172416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:33.172383 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:35.263935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:35.263897 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:35.440705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:35.440641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:37.572120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:37.572089 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:41.970017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:41.969984 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:42.829365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:42.829329 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:19:42.831596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:42.831568 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:19:46.435276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:46.435237 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:50.770759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:50.770724 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:55.155811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:55.155770 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:19:59.657705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:19:59.657667 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:01.108970 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:20:01.108928 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:20:02.871934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:02.871850 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:03.725854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:03.722928 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:05.273534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.273497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:05.498040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.498003 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:05.500528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.500505 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j"] Nov 01 15:20:05.550723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.550648 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-15ad1e-nqx8j"] Nov 01 15:20:05.674727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.674694 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v"] Nov 01 15:20:05.674932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.674892 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" containerID="cri-o://890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907" gracePeriod=30 Nov 01 15:20:05.745792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.745762 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:05.983829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.983749 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4"] Nov 01 15:20:05.989083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:05.989057 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.043178 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.043146 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4"] Nov 01 15:20:06.127864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.127829 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.127888 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.127998 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.128038 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.128211 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.128319 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.128382 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128470 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.128435 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq724\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.128507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.128468 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.229889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.229861 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.229889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.229892 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.229963 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230003 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230043 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230088 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq724\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230110 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230181 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230285 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230423 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230504 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230630 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230758 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.230984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.230966 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.232818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.232790 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.233025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.233007 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.286699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.286667 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.287042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.286983 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq724\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lbfq4\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.303280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.303259 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:06.719509 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:06.719476 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520cc1ca_a6c0_47ee_adf4_e288b0fb52d0.slice/crio-478231e2032f93bd9bf6e88ad63b590ab14c5d27a0c9357c216df78b713e53c9 WatchSource:0}: Error finding container 478231e2032f93bd9bf6e88ad63b590ab14c5d27a0c9357c216df78b713e53c9: Status 404 returned error can't find the container with id 478231e2032f93bd9bf6e88ad63b590ab14c5d27a0c9357c216df78b713e53c9 Nov 01 15:20:06.722638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.722598 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:20:06.722741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.722689 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:20:06.722741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.722731 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:20:06.723712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:06.723691 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4"] Nov 01 15:20:07.087503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.087463 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43915c8e-d254-4f89-88b3-715566829754" path="/var/lib/kubelet/pods/43915c8e-d254-4f89-88b3-715566829754/volumes" Nov 01 15:20:07.149593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.149553 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" event={"ID":"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0","Type":"ContainerStarted","Data":"0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a"} Nov 01 15:20:07.149800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.149618 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" event={"ID":"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0","Type":"ContainerStarted","Data":"478231e2032f93bd9bf6e88ad63b590ab14c5d27a0c9357c216df78b713e53c9"} Nov 01 15:20:07.268506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.268452 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podStartSLOduration=2.268436792 podStartE2EDuration="2.268436792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:07.265395354 +0000 UTC m=+3746.323547203" watchObservedRunningTime="2026-11-01 15:20:07.268436792 +0000 UTC m=+3746.326588642" Nov 01 15:20:07.307253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.307200 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:07.307938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.307914 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:07.308138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:07.308102 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:08.304553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.304511 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:08.304777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.304566 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:08.415670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.415638 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:08.590870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.590748 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:08.900932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.900829 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr"] Nov 01 15:20:08.905611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.905588 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:08.962368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.960061 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:20:08.962368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:08.961176 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr"] Nov 01 15:20:09.002950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.002918 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:20:09.075366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.075334 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.075559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.075475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.075559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.075549 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776kj\" (UniqueName: \"kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.176857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.176776 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.177024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.176935 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.177116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.177077 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776kj\" (UniqueName: \"kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.177596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.177571 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.179505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.179485 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.250135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.250098 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776kj\" (UniqueName: \"kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj\") pod \"maas-api-revoke-sub-ab96954c-6xclr\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.304522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.304486 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:09.304729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.304540 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:09.519783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.519753 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:09.560036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.559989 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:09.964461 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:09.964430 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8422c6ce_23b8_4ab1_89dc_10d869fccfb9.slice/crio-6532b09ffc7938d66a42563f3245433f4b49d3d93f5e97ac6d302a1a5ce6cf3e WatchSource:0}: Error finding container 6532b09ffc7938d66a42563f3245433f4b49d3d93f5e97ac6d302a1a5ce6cf3e: Status 404 returned error can't find the container with id 6532b09ffc7938d66a42563f3245433f4b49d3d93f5e97ac6d302a1a5ce6cf3e Nov 01 15:20:09.971374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:09.971345 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr"] Nov 01 15:20:10.175111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:10.175067 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" event={"ID":"8422c6ce-23b8-4ab1-89dc-10d869fccfb9","Type":"ContainerStarted","Data":"6532b09ffc7938d66a42563f3245433f4b49d3d93f5e97ac6d302a1a5ce6cf3e"} Nov 01 15:20:10.303825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:10.303789 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:10.304006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:10.303842 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:10.992475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:10.992447 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:20:11.153018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.152925 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.152974 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153013 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153064 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153096 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153315 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:11.153520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153344 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153390 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:20:11.153520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153395 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153485 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153517 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn\") pod \"d75df714-1cda-42e8-a61f-da2de949f015\" (UID: \"d75df714-1cda-42e8-a61f-da2de949f015\") " Nov 01 15:20:11.153873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153613 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:11.153873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153673 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:11.153873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.153744 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:11.154192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.154168 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75df714-1cda-42e8-a61f-da2de949f015-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.154311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.154197 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.154311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.154249 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.154311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.154265 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.154311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.154280 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.155630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.155596 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:20:11.155862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.155836 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "kube-api-access-wb5fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:11.155959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.155909 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:11.155959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.155912 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy" pod "d75df714-1cda-42e8-a61f-da2de949f015" (UID: "d75df714-1cda-42e8-a61f-da2de949f015"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:11.194411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.194378 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" event={"ID":"8422c6ce-23b8-4ab1-89dc-10d869fccfb9","Type":"ContainerStarted","Data":"9e1b80c7bba921219d5c6aa8df2ab820b457041a512b955702fa4d06cb88f71b"} Nov 01 15:20:11.196516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.196488 5473 generic.go:356] "Generic (PLEG): container finished" podID="d75df714-1cda-42e8-a61f-da2de949f015" containerID="890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907" exitCode=0 Nov 01 15:20:11.196760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.196539 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" Nov 01 15:20:11.196890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.196555 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" event={"ID":"d75df714-1cda-42e8-a61f-da2de949f015","Type":"ContainerDied","Data":"890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907"} Nov 01 15:20:11.196975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.196872 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" event={"ID":"d75df714-1cda-42e8-a61f-da2de949f015","Type":"ContainerDied","Data":"b9c4e4f760a15f8347819d7edc3a1686b088d1e2ffe142d2bf83d8de647d3e38"} Nov 01 15:20:11.196975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.196924 5473 scope.go:122] "RemoveContainer" containerID="890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907" Nov 01 15:20:11.255436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.255393 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.255616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.255502 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75df714-1cda-42e8-a61f-da2de949f015-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.255616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.255520 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75df714-1cda-42e8-a61f-da2de949f015-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.255616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.255531 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb5fn\" (UniqueName: \"kubernetes.io/projected/d75df714-1cda-42e8-a61f-da2de949f015-kube-api-access-wb5fn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:11.278390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.270483 5473 scope.go:122] "RemoveContainer" containerID="caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4" Nov 01 15:20:11.291841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.291815 5473 scope.go:122] "RemoveContainer" containerID="890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907" Nov 01 15:20:11.292181 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:20:11.292154 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907\": container with ID starting with 890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907 not found: ID does not exist" containerID="890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907" Nov 01 15:20:11.292304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.292194 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907"} err="failed to get container status \"890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907\": rpc error: code = NotFound desc = could not find container \"890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907\": container with ID starting with 890433b3ee6e09af6834ba08131afbed200daf009d9353d84bfd73bed20de907 not found: ID does not exist" Nov 01 15:20:11.292304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.292235 5473 scope.go:122] "RemoveContainer" containerID="caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4" Nov 01 15:20:11.292517 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:20:11.292496 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4\": container with ID starting with caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4 not found: ID does not exist" containerID="caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4" Nov 01 15:20:11.292562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.292524 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4"} err="failed to get container status \"caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4\": rpc error: code = NotFound desc = could not find container \"caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4\": container with ID starting with caeff2cd3add94ef5f6cef8d745299cc57f1a5d2a756140d936fc836bc810ee4 not found: ID does not exist" Nov 01 15:20:11.304378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.304344 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:11.304503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.304395 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:11.350287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.350244 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:11.510152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.510067 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v"] Nov 01 15:20:11.518152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.518116 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v"] Nov 01 15:20:11.912306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.912263 5473 patch_prober.go:29] interesting pod/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:20:11.912500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:11.912324 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-15ad1e-openshift-default-5bdfccfcf7-sqj9v" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:20:12.207269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.207167 5473 generic.go:356] "Generic (PLEG): container finished" podID="8422c6ce-23b8-4ab1-89dc-10d869fccfb9" containerID="9e1b80c7bba921219d5c6aa8df2ab820b457041a512b955702fa4d06cb88f71b" exitCode=0 Nov 01 15:20:12.207269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.207261 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" event={"ID":"8422c6ce-23b8-4ab1-89dc-10d869fccfb9","Type":"ContainerDied","Data":"9e1b80c7bba921219d5c6aa8df2ab820b457041a512b955702fa4d06cb88f71b"} Nov 01 15:20:12.304439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.304402 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:12.304607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.304472 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:12.395386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.395361 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:12.470986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.470894 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca\") pod \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " Nov 01 15:20:12.470986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.470988 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj\" (UniqueName: \"kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj\") pod \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " Nov 01 15:20:12.471294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.471017 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca\") pod \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\" (UID: \"8422c6ce-23b8-4ab1-89dc-10d869fccfb9\") " Nov 01 15:20:12.471463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.471436 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca" pod "8422c6ce-23b8-4ab1-89dc-10d869fccfb9" (UID: "8422c6ce-23b8-4ab1-89dc-10d869fccfb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:20:12.471656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.471622 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:12.473274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.473249 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca" pod "8422c6ce-23b8-4ab1-89dc-10d869fccfb9" (UID: "8422c6ce-23b8-4ab1-89dc-10d869fccfb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:20:12.473374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.473268 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj" pod "8422c6ce-23b8-4ab1-89dc-10d869fccfb9" (UID: "8422c6ce-23b8-4ab1-89dc-10d869fccfb9"). InnerVolumeSpecName "kube-api-access-776kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:12.572652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.572610 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:12.572652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.572651 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776kj\" (UniqueName: \"kubernetes.io/projected/8422c6ce-23b8-4ab1-89dc-10d869fccfb9-kube-api-access-776kj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:12.581018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.580979 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:12.748933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:12.748827 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:13.082700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:13.082667 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75df714-1cda-42e8-a61f-da2de949f015" path="/var/lib/kubelet/pods/d75df714-1cda-42e8-a61f-da2de949f015/volumes" Nov 01 15:20:13.216087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:13.216058 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" Nov 01 15:20:13.216087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:13.216071 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr" event={"ID":"8422c6ce-23b8-4ab1-89dc-10d869fccfb9","Type":"ContainerDied","Data":"6532b09ffc7938d66a42563f3245433f4b49d3d93f5e97ac6d302a1a5ce6cf3e"} Nov 01 15:20:13.216604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:13.216125 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6532b09ffc7938d66a42563f3245433f4b49d3d93f5e97ac6d302a1a5ce6cf3e" Nov 01 15:20:13.304685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:13.304644 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:13.304892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:13.304712 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:14.304062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:14.304024 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" start-of-body= Nov 01 15:20:14.304494 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:14.304091 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": dial tcp 10.134.1.38:15021: connect: connection refused" Nov 01 15:20:15.451622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:15.451588 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:15.452232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:15.452161 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:15.453178 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:15.453155 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:20:15.918857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:15.918818 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:18.817697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:18.817660 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:18.993052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:18.993014 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:20.270154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:20.270118 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:20.518186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:20.518149 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:20.711688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:20.711588 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr"] Nov 01 15:20:20.712030 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:20.711968 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" containerID="cri-o://4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9" gracePeriod=30 Nov 01 15:20:20.737019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:20.736987 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:21.311906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:21.311866 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:21.507430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:21.507384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:21.966479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:21.966449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:22.422591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.422556 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58"] Nov 01 15:20:22.423197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423184 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" Nov 01 15:20:22.423271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423198 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" Nov 01 15:20:22.423271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423235 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" Nov 01 15:20:22.423271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423241 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" Nov 01 15:20:22.423271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423259 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8422c6ce-23b8-4ab1-89dc-10d869fccfb9" containerName="revoke-keys" Nov 01 15:20:22.423271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423264 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8422c6ce-23b8-4ab1-89dc-10d869fccfb9" containerName="revoke-keys" Nov 01 15:20:22.423449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423361 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" Nov 01 15:20:22.423449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.423371 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8422c6ce-23b8-4ab1-89dc-10d869fccfb9" containerName="revoke-keys" Nov 01 15:20:22.426894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.426878 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.480321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.480287 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fc8c64-openshift-default-dockercfg-rb8l5\"" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516078 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516159 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516256 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516305 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516348 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516380 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516475 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516542 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.518821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.516614 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4pg\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.553352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.553319 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:22.563297 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.563263 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58"] Nov 01 15:20:22.621998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.621963 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622022 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622058 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622081 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622505 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622534 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622584 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622588 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622631 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622659 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622710 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4pg\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622743 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.622888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.622777 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.623320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.623295 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.625279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.625256 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.625536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.625513 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.680911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.680838 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.681059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.680962 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4pg\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg\") pod \"e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.741656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.741623 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:22.804237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:22.804047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:23.244997 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:23.244967 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb94eef_c4a0_4f62_aa95_3b1b7f69f803.slice/crio-c7df585e0819f85f7d1d8262ff18c35f131de959333516dcc405281b8bcbba82 WatchSource:0}: Error finding container c7df585e0819f85f7d1d8262ff18c35f131de959333516dcc405281b8bcbba82: Status 404 returned error can't find the container with id c7df585e0819f85f7d1d8262ff18c35f131de959333516dcc405281b8bcbba82 Nov 01 15:20:23.247899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:23.247864 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:20:23.247990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:23.247933 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:20:23.247990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:23.247961 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:20:23.248719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:23.248693 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58"] Nov 01 15:20:23.308754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:23.308709 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" event={"ID":"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803","Type":"ContainerStarted","Data":"c7df585e0819f85f7d1d8262ff18c35f131de959333516dcc405281b8bcbba82"} Nov 01 15:20:24.318957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:24.318925 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" event={"ID":"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803","Type":"ContainerStarted","Data":"4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70"} Nov 01 15:20:24.445397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:24.445356 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:24.459489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:24.459440 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podStartSLOduration=2.4594264089999998 podStartE2EDuration="2.459426409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:24.455101611 +0000 UTC m=+3763.513253473" watchObservedRunningTime="2026-11-01 15:20:24.459426409 +0000 UTC m=+3763.517578259" Nov 01 15:20:24.616176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:24.616058 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:24.742201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:24.742165 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:24.749123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:24.749095 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:25.327211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:25.327179 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:25.328253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:25.328233 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:20:25.515556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:25.515511 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:25.992742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:25.992674 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:20:26.087061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087028 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087075 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087156 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087281 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087320 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087354 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087381 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087552 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:26.087610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087575 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:26.087610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087584 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087793 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087756 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket\") pod \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\" (UID: \"6b65ea53-8ec4-4ac8-8063-755d1f64faec\") " Nov 01 15:20:26.087856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087791 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:26.087905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.087840 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:20:26.088119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.088095 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:26.088460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.088432 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.088460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.088461 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.088656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.088476 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.088656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.088491 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.088656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.088514 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.089579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.089554 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:26.089686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.089609 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "kube-api-access-b4hd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:26.089686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.089643 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:20:26.089934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.089916 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy" pod "6b65ea53-8ec4-4ac8-8063-755d1f64faec" (UID: "6b65ea53-8ec4-4ac8-8063-755d1f64faec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:20:26.189633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.189577 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.189633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.189629 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.189901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.189644 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hd6\" (UniqueName: \"kubernetes.io/projected/6b65ea53-8ec4-4ac8-8063-755d1f64faec-kube-api-access-b4hd6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.189901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.189660 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b65ea53-8ec4-4ac8-8063-755d1f64faec-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:26.337054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.337018 5473 generic.go:356] "Generic (PLEG): container finished" podID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerID="4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9" exitCode=0 Nov 01 15:20:26.337529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.337078 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" Nov 01 15:20:26.337529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.337092 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" event={"ID":"6b65ea53-8ec4-4ac8-8063-755d1f64faec","Type":"ContainerDied","Data":"4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9"} Nov 01 15:20:26.337529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.337143 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr" event={"ID":"6b65ea53-8ec4-4ac8-8063-755d1f64faec","Type":"ContainerDied","Data":"4d1635dde60f7632c12a92ffc19f25da72b3bf678174306d945988af77493cc1"} Nov 01 15:20:26.337529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.337163 5473 scope.go:122] "RemoveContainer" containerID="4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9" Nov 01 15:20:26.349365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.349344 5473 scope.go:122] "RemoveContainer" containerID="514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910" Nov 01 15:20:26.359166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.359141 5473 scope.go:122] "RemoveContainer" containerID="4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9" Nov 01 15:20:26.359522 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:20:26.359497 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9\": container with ID starting with 4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9 not found: ID does not exist" containerID="4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9" Nov 01 15:20:26.359603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.359526 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9"} err="failed to get container status \"4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9\": rpc error: code = NotFound desc = could not find container \"4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9\": container with ID starting with 4f3ffbefb47756c183cc223f1b06fdd3a9ed6f972c9d82e4f6638ae6b8ee08b9 not found: ID does not exist" Nov 01 15:20:26.359603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.359544 5473 scope.go:122] "RemoveContainer" containerID="514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910" Nov 01 15:20:26.359808 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:20:26.359792 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910\": container with ID starting with 514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910 not found: ID does not exist" containerID="514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910" Nov 01 15:20:26.359852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.359810 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910"} err="failed to get container status \"514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910\": rpc error: code = NotFound desc = could not find container \"514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910\": container with ID starting with 514ea0beed44d976516cef72b3aca9e4c6235b95a8f29d42afdcfaae7b978910 not found: ID does not exist" Nov 01 15:20:26.508336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.508289 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr"] Nov 01 15:20:26.557381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.557350 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-928a13-openshift-default-79b4fbf-bqcjr"] Nov 01 15:20:26.771735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.771696 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r"] Nov 01 15:20:26.772410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.772394 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" Nov 01 15:20:26.772481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.772411 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" Nov 01 15:20:26.772481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.772441 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" Nov 01 15:20:26.772481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.772447 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" Nov 01 15:20:26.772586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.772541 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75df714-1cda-42e8-a61f-da2de949f015" containerName="istio-proxy" Nov 01 15:20:26.772586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.772553 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" Nov 01 15:20:26.776371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.776354 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829369 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wm2wx\"" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829375 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-dda3bf67b7e4\"" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829413 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829403 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-287c3706f19e\"" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829369 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4222812391bf\"" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829447 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 15:20:26.829474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829468 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r"] Nov 01 15:20:26.829944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829488 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 15:20:26.829944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.829472 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 15:20:26.899709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.899675 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:26.899869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.899730 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:26.899869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.899819 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6774\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:26.899969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:26.899916 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.001560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.001525 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.001737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.001592 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.001737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.001632 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6774\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.001737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.001715 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.002391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.002372 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.002452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.002393 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.004426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.004402 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.059397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.059331 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6774\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774\") pod \"payload-processing-external-model-75b899bcb5-wdx6r\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.083421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.083394 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" path="/var/lib/kubelet/pods/6b65ea53-8ec4-4ac8-8063-755d1f64faec/volumes" Nov 01 15:20:27.090874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.090848 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:27.202570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.202541 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:27.459262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.459134 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:27.469006 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:27.468957 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370dc879_b433_4c3a_8752_027dcc336131.slice/crio-47f5a982b82006a1519d678b9dee1a678a10b6d298af626ad8e2f56cb615e752 WatchSource:0}: Error finding container 47f5a982b82006a1519d678b9dee1a678a10b6d298af626ad8e2f56cb615e752: Status 404 returned error can't find the container with id 47f5a982b82006a1519d678b9dee1a678a10b6d298af626ad8e2f56cb615e752 Nov 01 15:20:27.472449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.472423 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r"] Nov 01 15:20:27.734547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:27.734462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:28.368691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:28.368651 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" event={"ID":"370dc879-b433-4c3a-8752-027dcc336131","Type":"ContainerStarted","Data":"962153d2167b88d3ce92ac0808d3e4318909ecd7fb2b92da2dbed8c77efe017a"} Nov 01 15:20:28.368898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:28.368744 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:28.368898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:28.368766 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" event={"ID":"370dc879-b433-4c3a-8752-027dcc336131","Type":"ContainerStarted","Data":"47f5a982b82006a1519d678b9dee1a678a10b6d298af626ad8e2f56cb615e752"} Nov 01 15:20:28.486183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:28.486131 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" podStartSLOduration=2.486115746 podStartE2EDuration="2.486115746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:28.482388618 +0000 UTC m=+3767.540540468" watchObservedRunningTime="2026-11-01 15:20:28.486115746 +0000 UTC m=+3767.544267596" Nov 01 15:20:29.051080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:29.051045 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw"] Nov 01 15:20:29.065634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:29.065599 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bd84c1-wp9pw"] Nov 01 15:20:29.082313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:29.082284 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7644595-12fe-48ac-92c8-765cc57d4a4c" path="/var/lib/kubelet/pods/a7644595-12fe-48ac-92c8-765cc57d4a4c/volumes" Nov 01 15:20:30.319187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:30.319151 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:30.575977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:30.575886 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:30.749237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:30.748868 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:31.108201 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:20:31.108155 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:20:31.884891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:31.884850 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:32.057601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.057562 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:32.437387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.437352 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn"] Nov 01 15:20:32.441518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.438542 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b65ea53-8ec4-4ac8-8063-755d1f64faec" containerName="istio-proxy" Nov 01 15:20:32.455249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.454342 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:32.495666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.495634 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn"] Nov 01 15:20:32.541709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.541673 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fc8c64-dockercfg-n59s9\"" Nov 01 15:20:32.541893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.541730 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fc8c64\"" Nov 01 15:20:32.623560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.623527 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:32.623763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.623688 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjl2\" (UniqueName: \"kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2\") pod \"payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:32.725531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.725440 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjl2\" (UniqueName: \"kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2\") pod \"payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:32.725690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.725647 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:32.726374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.726354 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:32.783068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:32.783040 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjl2\" (UniqueName: \"kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2\") pod \"payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:33.077202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:33.077169 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:33.451083 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:33.451049 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703dd976_3097_496f_a5e5_dec14a920b09.slice/crio-e7569e8583c935fbfa783c5236f8ed3aef2e299b776c8c001621b94426d2eefc WatchSource:0}: Error finding container e7569e8583c935fbfa783c5236f8ed3aef2e299b776c8c001621b94426d2eefc: Status 404 returned error can't find the container with id e7569e8583c935fbfa783c5236f8ed3aef2e299b776c8c001621b94426d2eefc Nov 01 15:20:33.455813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:33.455787 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn"] Nov 01 15:20:33.691494 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:33.691459 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:34.089159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.089126 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r"] Nov 01 15:20:34.092229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.092188 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:34.354359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.354280 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:34.470238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.469983 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" event={"ID":"703dd976-3097-496f-a5e5-dec14a920b09","Type":"ContainerStarted","Data":"a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0"} Nov 01 15:20:34.470238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.470037 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" event={"ID":"703dd976-3097-496f-a5e5-dec14a920b09","Type":"ContainerStarted","Data":"e7569e8583c935fbfa783c5236f8ed3aef2e299b776c8c001621b94426d2eefc"} Nov 01 15:20:34.470507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.470314 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:34.591263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.591184 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" podStartSLOduration=2.591167293 podStartE2EDuration="2.591167293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:34.58638058 +0000 UTC m=+3773.644532429" watchObservedRunningTime="2026-11-01 15:20:34.591167293 +0000 UTC m=+3773.649319142" Nov 01 15:20:34.608139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.608051 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:34.880207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:34.880130 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:39.053194 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:39.053152 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:39.090379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:39.090335 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" podUID="370dc879-b433-4c3a-8752-027dcc336131" containerName="payload-processing" containerID="cri-o://962153d2167b88d3ce92ac0808d3e4318909ecd7fb2b92da2dbed8c77efe017a" gracePeriod=25 Nov 01 15:20:39.293255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:39.290756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:39.524300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:39.483859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:39.758365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:39.720332 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:40.166562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.166527 5473 generic.go:356] "Generic (PLEG): container finished" podID="370dc879-b433-4c3a-8752-027dcc336131" containerID="962153d2167b88d3ce92ac0808d3e4318909ecd7fb2b92da2dbed8c77efe017a" exitCode=0 Nov 01 15:20:40.182421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.174471 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" event={"ID":"370dc879-b433-4c3a-8752-027dcc336131","Type":"ContainerDied","Data":"962153d2167b88d3ce92ac0808d3e4318909ecd7fb2b92da2dbed8c77efe017a"} Nov 01 15:20:40.182421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.182385 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt"] Nov 01 15:20:40.314243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.250869 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd2db9c-5sbmt"] Nov 01 15:20:40.314509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.279657 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:40.314711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.283597 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:40.384747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.383580 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774\") pod \"370dc879-b433-4c3a-8752-027dcc336131\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " Nov 01 15:20:40.384747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.383692 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials\") pod \"370dc879-b433-4c3a-8752-027dcc336131\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " Nov 01 15:20:40.384747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.383738 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume\") pod \"370dc879-b433-4c3a-8752-027dcc336131\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " Nov 01 15:20:40.384747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.383831 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume\") pod \"370dc879-b433-4c3a-8752-027dcc336131\" (UID: \"370dc879-b433-4c3a-8752-027dcc336131\") " Nov 01 15:20:40.385363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.385337 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume" pod "370dc879-b433-4c3a-8752-027dcc336131" (UID: "370dc879-b433-4c3a-8752-027dcc336131"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:20:40.403517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.403489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume" pod "370dc879-b433-4c3a-8752-027dcc336131" (UID: "370dc879-b433-4c3a-8752-027dcc336131"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:20:40.457466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.445574 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774" pod "370dc879-b433-4c3a-8752-027dcc336131" (UID: "370dc879-b433-4c3a-8752-027dcc336131"). InnerVolumeSpecName "kube-api-access-f6774". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:40.471640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.471605 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials" pod "370dc879-b433-4c3a-8752-027dcc336131" (UID: "370dc879-b433-4c3a-8752-027dcc336131"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:20:40.499240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.488287 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:40.499240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.488320 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6774\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-kube-api-access-f6774\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:40.499240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.488336 5473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/370dc879-b433-4c3a-8752-027dcc336131-provider-credentials\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:40.499240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.488350 5473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/370dc879-b433-4c3a-8752-027dcc336131-routing-overlay-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:20:40.683396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.667815 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:20:40.755358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:40.755317 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:20:41.225957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:41.225914 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe" path="/var/lib/kubelet/pods/b3bd3d1b-b9f4-4dd3-953c-ce2517277cfe/volumes" Nov 01 15:20:41.343725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:41.343690 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" event={"ID":"370dc879-b433-4c3a-8752-027dcc336131","Type":"ContainerDied","Data":"47f5a982b82006a1519d678b9dee1a678a10b6d298af626ad8e2f56cb615e752"} Nov 01 15:20:41.344031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:41.344001 5473 scope.go:122] "RemoveContainer" containerID="962153d2167b88d3ce92ac0808d3e4318909ecd7fb2b92da2dbed8c77efe017a" Nov 01 15:20:41.352247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:41.344339 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r" Nov 01 15:20:41.521370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:41.521339 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r"] Nov 01 15:20:41.572914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:41.572866 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75b899bcb5-wdx6r"] Nov 01 15:20:43.082972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:43.082937 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370dc879-b433-4c3a-8752-027dcc336131" path="/var/lib/kubelet/pods/370dc879-b433-4c3a-8752-027dcc336131/volumes" Nov 01 15:20:44.625149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:44.625120 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:45.479900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:45.479872 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:20:49.108930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:49.108896 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:50.497122 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.497037 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:50.662266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.662199 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:50.878208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.878179 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk"] Nov 01 15:20:50.878837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.878822 5473 cpu_manager.go:414] "containerMap: removing container" podUID="370dc879-b433-4c3a-8752-027dcc336131" containerName="payload-processing" Nov 01 15:20:50.878886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.878839 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370dc879-b433-4c3a-8752-027dcc336131" containerName="payload-processing" Nov 01 15:20:50.878980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.878971 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="370dc879-b433-4c3a-8752-027dcc336131" containerName="payload-processing" Nov 01 15:20:50.882589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.882574 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:50.942015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.941926 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk"] Nov 01 15:20:50.993676 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.993186 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm6nf\" (UniqueName: \"kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf\") pod \"payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:50.993676 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:50.993374 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:51.021610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.021569 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3d5a5b-dockercfg-qvhqz\"" Nov 01 15:20:51.022095 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.022069 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3d5a5b\"" Nov 01 15:20:51.084320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.084292 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl"] Nov 01 15:20:51.088563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.088543 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.095048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.095018 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6nf\" (UniqueName: \"kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf\") pod \"payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:51.095227 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.095195 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:51.095962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.095937 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:51.102246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.102199 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl"] Nov 01 15:20:51.152670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.152599 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6nf\" (UniqueName: \"kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf\") pod \"payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:51.197294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.197015 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtjk\" (UniqueName: \"kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk\") pod \"payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.197477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.197423 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.208351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.208328 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:51.299950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.298576 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtjk\" (UniqueName: \"kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk\") pod \"payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.299950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.298711 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.299950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.299831 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.355608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.355585 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtjk\" (UniqueName: \"kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk\") pod \"payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.405129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.405044 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:51.581706 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:51.581672 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b798657_7cea_4937_8332_f4c4bec1f369.slice/crio-be58de0e803e2a21678041520f8d9d0fee14f45d2bf1735e5d636f2a75c287b7 WatchSource:0}: Error finding container be58de0e803e2a21678041520f8d9d0fee14f45d2bf1735e5d636f2a75c287b7: Status 404 returned error can't find the container with id be58de0e803e2a21678041520f8d9d0fee14f45d2bf1735e5d636f2a75c287b7 Nov 01 15:20:51.586527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.586501 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk"] Nov 01 15:20:51.779048 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:20:51.779019 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657c687b_8dad_45e8_b9dc_fa07caacb5e2.slice/crio-bd299a18ab1e11145aa2eb8cb8bacb58e25fab10050589d16b99b4d031d1c081 WatchSource:0}: Error finding container bd299a18ab1e11145aa2eb8cb8bacb58e25fab10050589d16b99b4d031d1c081: Status 404 returned error can't find the container with id bd299a18ab1e11145aa2eb8cb8bacb58e25fab10050589d16b99b4d031d1c081 Nov 01 15:20:51.782668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:51.782646 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl"] Nov 01 15:20:52.435957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.435917 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" event={"ID":"657c687b-8dad-45e8-b9dc-fa07caacb5e2","Type":"ContainerStarted","Data":"3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51"} Nov 01 15:20:52.436186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.435984 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:20:52.436186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.436006 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" event={"ID":"657c687b-8dad-45e8-b9dc-fa07caacb5e2","Type":"ContainerStarted","Data":"bd299a18ab1e11145aa2eb8cb8bacb58e25fab10050589d16b99b4d031d1c081"} Nov 01 15:20:52.437641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.437614 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" event={"ID":"3b798657-7cea-4937-8332-f4c4bec1f369","Type":"ContainerStarted","Data":"2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3"} Nov 01 15:20:52.437744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.437658 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" event={"ID":"3b798657-7cea-4937-8332-f4c4bec1f369","Type":"ContainerStarted","Data":"be58de0e803e2a21678041520f8d9d0fee14f45d2bf1735e5d636f2a75c287b7"} Nov 01 15:20:52.437744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.437684 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:20:52.550206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.550154 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" podStartSLOduration=1.550142271 podStartE2EDuration="1.550142271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:52.549731825 +0000 UTC m=+3791.607883677" watchObservedRunningTime="2026-11-01 15:20:52.550142271 +0000 UTC m=+3791.608294154" Nov 01 15:20:52.667134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:52.667067 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" podStartSLOduration=2.667008832 podStartE2EDuration="2.667008832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:20:52.662772992 +0000 UTC m=+3791.720924843" watchObservedRunningTime="2026-11-01 15:20:52.667008832 +0000 UTC m=+3791.725160685" Nov 01 15:20:53.583746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:53.583707 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:20:58.022727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:20:58.022695 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:01.110838 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:21:01.110800 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:21:02.382317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:02.382278 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:03.194395 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:03.194358 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:03.448147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:03.448060 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:21:03.448147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:03.448130 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:21:06.807622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:06.807584 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:11.242005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:11.241967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:15.599144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:15.599104 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:20.109363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:20.109324 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:22.467335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:22.467290 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Nov 01 15:21:22.478357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:22.478288 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:21:22.546421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:22.546364 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:21:22.622344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:22.622304 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:21:24.885850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:24.885816 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:29.290768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:29.290730 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:31.107867 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:21:31.107828 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:21:33.835999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:33.835937 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:38.066594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:38.066562 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:38.239054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:38.239017 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:38.404302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:38.404202 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:42.662097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:42.662061 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:44.059628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:44.059596 5473 scope.go:122] "RemoveContainer" containerID="be356586e792ef4aa9a0b433829f81e35e5cf3ac10009b4ea48cd630ccd9d9b4" Nov 01 15:21:44.070467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:44.070449 5473 scope.go:122] "RemoveContainer" containerID="8a178274fae497b98af92a5e15d74595c90e7d3c612d4e3ad4fa831845c83cf8" Nov 01 15:21:47.213176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:47.213137 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:51.592455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:51.592419 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:55.972187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:55.972143 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:56.206274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:56.206242 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:21:56.370616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:21:56.370580 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:00.500263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:00.471331 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:00.500263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:00.480043 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:00.514234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:00.505774 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:00.586248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:00.523416 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:00.717240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:00.716285 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:00.859239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:00.858501 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:01.263930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.263535 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:22:01.263930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.263625 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.269437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.269407 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:22:01.269744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.269716 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.305009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.304976 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:22:01.305733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.305686 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.346699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.346317 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.385077 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:22:01.385040 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:22:01.414826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.414507 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:01.489248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.475439 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.489248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.475883 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 15:22:01.489248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.487306 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.489248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.487712 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 15:22:01.523295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.512306 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.523295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.512841 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 15:22:01.523295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.516366 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:22:01.523295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.516413 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:01.525325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.524096 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 01 15:22:01.525325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.524521 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 15:22:01.579788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.579573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:01.589051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:01.588294 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:22:02.098978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:02.098947 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 15:22:02.230257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:02.229680 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" Nov 01 15:22:02.275081 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:02.271480 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 15:22:02.279568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:02.279505 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 15:22:03.650052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:03.650017 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:05.503100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:05.503070 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:09.145872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.145838 5473 generic.go:356] "Generic (PLEG): container finished" podID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerID="4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70" exitCode=0 Nov 01 15:22:09.146412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.145911 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" event={"ID":"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803","Type":"ContainerDied","Data":"4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70"} Nov 01 15:22:09.146412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.146367 5473 scope.go:122] "RemoveContainer" containerID="4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70" Nov 01 15:22:09.147921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.147892 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:22:09.148038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.147963 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:22:09.148038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.148001 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:22:09.666709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.666674 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:09.937900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.937823 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs"] Nov 01 15:22:09.942427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.942404 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:09.995506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:09.995480 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:22:10.000458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.000430 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs"] Nov 01 15:22:10.053532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.053496 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:22:10.086348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.086315 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9p4\" (UniqueName: \"kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.086527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.086474 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.086527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.086494 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.158732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.158696 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" event={"ID":"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803","Type":"ContainerStarted","Data":"6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3"} Nov 01 15:22:10.188003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.187912 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.188003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.187952 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.188263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.188158 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9p4\" (UniqueName: \"kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.188664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.188643 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.190427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.190407 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.254875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.254802 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9p4\" (UniqueName: \"kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4\") pod \"maas-api-revoke-sub-6b0aa222-r8lqs\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.257408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.257383 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:10.357447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.357350 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:22:10.359164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.359128 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:10.655896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.655860 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs"] Nov 01 15:22:10.746241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.743948 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:22:10.746241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.744093 5473 patch_prober.go:29] interesting pod/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 01 15:22:10.746241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.744138 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 01 15:22:10.782232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:10.782197 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:11.168419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.168381 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" event={"ID":"3e2d2507-57d8-43f8-8837-317145332ec3","Type":"ContainerStarted","Data":"8ed1cc9ed1e7a99656aaca19b7d7193ee7e2912f6a9ea0da03381d1f831119d8"} Nov 01 15:22:11.168835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.168449 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" event={"ID":"3e2d2507-57d8-43f8-8837-317145332ec3","Type":"ContainerStarted","Data":"83c6cf5d205970ebba620792b286c4c2ecc2fb8d31126c14280b782d98862894"} Nov 01 15:22:11.222535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.222481 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:11.287017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.286968 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" podStartSLOduration=2.286953919 podStartE2EDuration="2.286953919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:11.285971799 +0000 UTC m=+3870.344123649" watchObservedRunningTime="2026-11-01 15:22:11.286953919 +0000 UTC m=+3870.345105818" Nov 01 15:22:11.484416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.484273 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:11.652606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.652573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:11.752912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.752676 5473 patch_prober.go:29] interesting pod/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 01 15:22:11.752912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:11.752743 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 01 15:22:12.177341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:12.177309 5473 generic.go:356] "Generic (PLEG): container finished" podID="3e2d2507-57d8-43f8-8837-317145332ec3" containerID="8ed1cc9ed1e7a99656aaca19b7d7193ee7e2912f6a9ea0da03381d1f831119d8" exitCode=0 Nov 01 15:22:12.177821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:12.177403 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" event={"ID":"3e2d2507-57d8-43f8-8837-317145332ec3","Type":"ContainerDied","Data":"8ed1cc9ed1e7a99656aaca19b7d7193ee7e2912f6a9ea0da03381d1f831119d8"} Nov 01 15:22:12.276875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:12.276825 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:12.480044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:12.479943 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:12.749296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:12.743668 5473 patch_prober.go:29] interesting pod/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 01 15:22:12.749296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:12.743730 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 01 15:22:13.388021 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.387993 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:13.470048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.469966 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca\") pod \"3e2d2507-57d8-43f8-8837-317145332ec3\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " Nov 01 15:22:13.470048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.470014 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca\") pod \"3e2d2507-57d8-43f8-8837-317145332ec3\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " Nov 01 15:22:13.470299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.470077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4\" (UniqueName: \"kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4\") pod \"3e2d2507-57d8-43f8-8837-317145332ec3\" (UID: \"3e2d2507-57d8-43f8-8837-317145332ec3\") " Nov 01 15:22:13.470442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.470414 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca" pod "3e2d2507-57d8-43f8-8837-317145332ec3" (UID: "3e2d2507-57d8-43f8-8837-317145332ec3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:22:13.470544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.470529 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:22:13.472187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.472161 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca" pod "3e2d2507-57d8-43f8-8837-317145332ec3" (UID: "3e2d2507-57d8-43f8-8837-317145332ec3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:22:13.472289 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.472263 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4" pod "3e2d2507-57d8-43f8-8837-317145332ec3" (UID: "3e2d2507-57d8-43f8-8837-317145332ec3"). InnerVolumeSpecName "kube-api-access-6n9p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:22:13.571536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.571493 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e2d2507-57d8-43f8-8837-317145332ec3-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:22:13.571536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.571527 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9p4\" (UniqueName: \"kubernetes.io/projected/3e2d2507-57d8-43f8-8837-317145332ec3-kube-api-access-6n9p4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:22:13.749295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.748318 5473 patch_prober.go:29] interesting pod/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 01 15:22:13.749295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:13.748383 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 01 15:22:14.194106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.194068 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" Nov 01 15:22:14.194335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.194100 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs" event={"ID":"3e2d2507-57d8-43f8-8837-317145332ec3","Type":"ContainerDied","Data":"83c6cf5d205970ebba620792b286c4c2ecc2fb8d31126c14280b782d98862894"} Nov 01 15:22:14.194335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.194153 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c6cf5d205970ebba620792b286c4c2ecc2fb8d31126c14280b782d98862894" Nov 01 15:22:14.749383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.749341 5473 patch_prober.go:29] interesting pod/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body= Nov 01 15:22:14.749827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.749409 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" Nov 01 15:22:14.841691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.841643 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:14.993661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:14.993630 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:15.747171 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:15.747142 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:22:15.748341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:15.748320 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:22:19.306555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:19.306456 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:22.009434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:22.009359 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:22.227495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:22.227462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:23.924813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:23.924774 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:24.089253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:24.089223 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:24.576866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:24.576830 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:24.768901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:24.768856 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:24.990473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:24.990398 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:25.354667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:25.354616 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:28.547588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:28.547557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:29.196366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:29.196337 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:29.389839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:29.389801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:31.110169 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:22:31.110132 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:22:31.119544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:31.119521 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:31.296808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:31.296775 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:32.273950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.273919 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:32.443716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.443682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:32.837274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.820510 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t"] Nov 01 15:22:32.837274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.821629 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3e2d2507-57d8-43f8-8837-317145332ec3" containerName="revoke-keys" Nov 01 15:22:32.837274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.821656 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e2d2507-57d8-43f8-8837-317145332ec3" containerName="revoke-keys" Nov 01 15:22:32.837274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.821838 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e2d2507-57d8-43f8-8837-317145332ec3" containerName="revoke-keys" Nov 01 15:22:32.889096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.889056 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t"] Nov 01 15:22:32.889278 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.889244 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:32.997908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.997880 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-775095\"" Nov 01 15:22:32.997908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:32.997915 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-775095-dockercfg-4kmsc\"" Nov 01 15:22:33.001667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.001644 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8"] Nov 01 15:22:33.008444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.008426 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.054790 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.054766 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.054921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.054888 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vz9f\" (UniqueName: \"kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f\") pod \"payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.055000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.054986 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.055175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.055157 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzbn\" (UniqueName: \"kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn\") pod \"payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.061340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.061318 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8"] Nov 01 15:22:33.156692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.156615 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzbn\" (UniqueName: \"kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn\") pod \"payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.156692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.156655 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.156887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.156708 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vz9f\" (UniqueName: \"kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f\") pod \"payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.156887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.156773 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.157502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.157482 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.157553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.157530 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.214631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.214598 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzbn\" (UniqueName: \"kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn\") pod \"payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.214773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.214691 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vz9f\" (UniqueName: \"kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f\") pod \"payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.323393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.323368 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:33.463000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.462930 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:33.512402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.512367 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:33.714481 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:22:33.714419 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4586c649_8a0a_4542_8fbf_7e7d7dc67fb0.slice/crio-893eca83e2e615cbde25b1f19a3990476205ef6f47615782ce733a5f6f54aca4 WatchSource:0}: Error finding container 893eca83e2e615cbde25b1f19a3990476205ef6f47615782ce733a5f6f54aca4: Status 404 returned error can't find the container with id 893eca83e2e615cbde25b1f19a3990476205ef6f47615782ce733a5f6f54aca4 Nov 01 15:22:33.718208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.718179 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8"] Nov 01 15:22:33.887483 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:22:33.887449 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bbb16e_8b84_4e34_b221_f04ba8dc51a2.slice/crio-f4fe2c6403c206c0e01805227566a5313ea6eb1d66374a46bc7eb7ebfe3f6e01 WatchSource:0}: Error finding container f4fe2c6403c206c0e01805227566a5313ea6eb1d66374a46bc7eb7ebfe3f6e01: Status 404 returned error can't find the container with id f4fe2c6403c206c0e01805227566a5313ea6eb1d66374a46bc7eb7ebfe3f6e01 Nov 01 15:22:33.889673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:33.889650 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t"] Nov 01 15:22:34.373581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.373541 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" event={"ID":"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0","Type":"ContainerStarted","Data":"986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc"} Nov 01 15:22:34.374253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.373607 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" event={"ID":"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0","Type":"ContainerStarted","Data":"893eca83e2e615cbde25b1f19a3990476205ef6f47615782ce733a5f6f54aca4"} Nov 01 15:22:34.374253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.373767 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:34.375316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.375296 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" event={"ID":"63bbb16e-8b84-4e34-b221-f04ba8dc51a2","Type":"ContainerStarted","Data":"eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797"} Nov 01 15:22:34.375403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.375341 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" event={"ID":"63bbb16e-8b84-4e34-b221-f04ba8dc51a2","Type":"ContainerStarted","Data":"f4fe2c6403c206c0e01805227566a5313ea6eb1d66374a46bc7eb7ebfe3f6e01"} Nov 01 15:22:34.375403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.375365 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:34.491121 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.491063 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" podStartSLOduration=2.491046325 podStartE2EDuration="2.491046325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:34.487841677 +0000 UTC m=+3893.545993526" watchObservedRunningTime="2026-11-01 15:22:34.491046325 +0000 UTC m=+3893.549198178" Nov 01 15:22:34.606751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:34.606685 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" podStartSLOduration=2.606665965 podStartE2EDuration="2.606665965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:34.601728951 +0000 UTC m=+3893.659880802" watchObservedRunningTime="2026-11-01 15:22:34.606665965 +0000 UTC m=+3893.664817816" Nov 01 15:22:36.918990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:36.918955 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:37.157077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.157047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:37.341636 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.341602 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:37.861595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.861565 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9"] Nov 01 15:22:37.866231 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.866193 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.892868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.892837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:37.913172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913138 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913186 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913348 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913411 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913583 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913645 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913626 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913744 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913779 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.913875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.913832 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:37.922778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.922751 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5feb76-openshift-default-dockercfg-xj8gw\"" Nov 01 15:22:37.923188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:37.923121 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9"] Nov 01 15:22:38.015075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015041 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015093 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015146 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015172 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015234 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015352 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015378 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015428 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.015558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.015483 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.017966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.016713 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.017966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.017462 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.017966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.017696 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.017966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.017920 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.018922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.018897 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.025289 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.024825 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.055353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.044547 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.061313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.059463 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:38.094728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.093957 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.094728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.094649 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b\") pod \"e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.180781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.180698 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:38.477829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.477726 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:38.602033 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:22:38.602005 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2940993e_3d29_49ca_b4bf_01f2fd2a88f5.slice/crio-9d1ffea550d86ff770339ccf1e99b712cbeebea348f7a1aa41cdc92ebee5c050 WatchSource:0}: Error finding container 9d1ffea550d86ff770339ccf1e99b712cbeebea348f7a1aa41cdc92ebee5c050: Status 404 returned error can't find the container with id 9d1ffea550d86ff770339ccf1e99b712cbeebea348f7a1aa41cdc92ebee5c050 Nov 01 15:22:38.605133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.605100 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:22:38.605243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.605169 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:22:38.605243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.605199 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:22:38.606581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:38.606553 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9"] Nov 01 15:22:39.413195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:39.413162 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" event={"ID":"2940993e-3d29-49ca-b4bf-01f2fd2a88f5","Type":"ContainerStarted","Data":"f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63"} Nov 01 15:22:39.413601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:39.413234 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" event={"ID":"2940993e-3d29-49ca-b4bf-01f2fd2a88f5","Type":"ContainerStarted","Data":"9d1ffea550d86ff770339ccf1e99b712cbeebea348f7a1aa41cdc92ebee5c050"} Nov 01 15:22:39.580299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:39.580248 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podStartSLOduration=2.580202282 podStartE2EDuration="2.580202282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:22:39.579610787 +0000 UTC m=+3898.637762634" watchObservedRunningTime="2026-11-01 15:22:39.580202282 +0000 UTC m=+3898.638354131" Nov 01 15:22:40.181185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:40.181149 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:40.186971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:40.186945 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:40.421034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:40.420990 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:40.422090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:40.422063 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:22:42.313176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:42.313134 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:42.498690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:42.498575 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:42.702991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:42.702840 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:44.435358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:44.435330 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:22:44.441669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:44.441648 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:22:45.385695 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:45.385661 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:22:45.385883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:45.385784 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:22:46.095262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:46.095202 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:46.257728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:46.257696 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:47.660895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:47.660863 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:52.027715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:52.027684 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:22:56.552773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:22:56.552724 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:00.989914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:00.989878 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:01.109009 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:23:01.108978 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:23:03.827705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:03.827671 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:05.501360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:05.501324 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:10.397555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:10.397520 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:10.576717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:10.576680 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:10.979337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:10.979304 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:11.606047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:11.605973 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:11.886666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:11.886559 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:15.012307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:15.012249 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:15.653892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:15.653853 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:15.827450 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:15.827377 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:18.801510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:18.801443 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:18.968637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:18.968605 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:19.169655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.169578 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n"] Nov 01 15:23:19.173900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.173884 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.230661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.230630 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n"] Nov 01 15:23:19.313073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.313031 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzk8\" (UniqueName: \"kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8\") pod \"payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.313453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.313424 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.365033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.364991 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89e5ef\"" Nov 01 15:23:19.365500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.365482 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-89e5ef-dockercfg-cwhrs\"" Nov 01 15:23:19.414827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.414783 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzk8\" (UniqueName: \"kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8\") pod \"payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.415021 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.414861 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.416011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.415984 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.471956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.471889 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzk8\" (UniqueName: \"kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8\") pod \"payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.488249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.488194 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:19.810149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.810118 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:19.874153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:19.868294 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n"] Nov 01 15:23:19.874153 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:23:19.873781 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a7a7d1_2155_40bd_8866_66924764d547.slice/crio-85ce3d5e542056e363c7ff18d9a1d8b4e8b1a093de97fb56990e670e5976e0b8 WatchSource:0}: Error finding container 85ce3d5e542056e363c7ff18d9a1d8b4e8b1a093de97fb56990e670e5976e0b8: Status 404 returned error can't find the container with id 85ce3d5e542056e363c7ff18d9a1d8b4e8b1a093de97fb56990e670e5976e0b8 Nov 01 15:23:20.793172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:20.793134 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" event={"ID":"21a7a7d1-2155-40bd-8866-66924764d547","Type":"ContainerStarted","Data":"ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6"} Nov 01 15:23:20.793405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:20.793195 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:20.793405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:20.793212 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" event={"ID":"21a7a7d1-2155-40bd-8866-66924764d547","Type":"ContainerStarted","Data":"85ce3d5e542056e363c7ff18d9a1d8b4e8b1a093de97fb56990e670e5976e0b8"} Nov 01 15:23:20.910372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:20.910321 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" podStartSLOduration=1.9103067089999999 podStartE2EDuration="1.910306709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:20.907381153 +0000 UTC m=+3939.965533002" watchObservedRunningTime="2026-11-01 15:23:20.910306709 +0000 UTC m=+3939.968458558" Nov 01 15:23:24.336979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:24.336944 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:27.431537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:27.431497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:27.850947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:27.850912 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:28.303331 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.303294 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758"] Nov 01 15:23:28.307867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.307849 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353166 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353287 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353349 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353394 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353459 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353484 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353507 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5jqw\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353549 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.355236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.353611 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.371377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.371341 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-04aa47-openshift-default-dockercfg-m8bsg\"" Nov 01 15:23:28.439691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.438751 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:28.453090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.453053 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758"] Nov 01 15:23:28.455519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.455488 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.455969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.455934 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.456155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.456138 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.456316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.456298 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.456541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.456524 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.456665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.456652 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.456947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.456927 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5jqw\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.457098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.457084 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.457271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.457255 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.457997 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.457974 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.458503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.458483 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.462517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.461581 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.462517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.461856 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.462517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.462132 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.463269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.463250 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.465850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.465826 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.530078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.529901 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.558393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.557770 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5jqw\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw\") pod \"e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.622939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.622901 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:28.687419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.686868 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:28.931517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:28.931423 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:29.057426 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:23:29.057400 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c919c2_feaf_4ae3_ba57_48c1d3f41261.slice/crio-f2ec0a96686d14ee07fea7e9ae634569726c69c0dfe9fbb27a396181e6438ae8 WatchSource:0}: Error finding container f2ec0a96686d14ee07fea7e9ae634569726c69c0dfe9fbb27a396181e6438ae8: Status 404 returned error can't find the container with id f2ec0a96686d14ee07fea7e9ae634569726c69c0dfe9fbb27a396181e6438ae8 Nov 01 15:23:29.061004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.060962 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:23:29.061093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.061055 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:23:29.061137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.061098 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:23:29.062612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.062592 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758"] Nov 01 15:23:29.863202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.863162 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" event={"ID":"d0c919c2-feaf-4ae3-ba57-48c1d3f41261","Type":"ContainerStarted","Data":"fea23a62621cffcb19105fef56ace4319276b434eb71ad1a6ebcdaf0d07c5e87"} Nov 01 15:23:29.863677 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.863237 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" event={"ID":"d0c919c2-feaf-4ae3-ba57-48c1d3f41261","Type":"ContainerStarted","Data":"f2ec0a96686d14ee07fea7e9ae634569726c69c0dfe9fbb27a396181e6438ae8"} Nov 01 15:23:29.984283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:29.984229 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podStartSLOduration=1.984203672 podStartE2EDuration="1.984203672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:29.980684202 +0000 UTC m=+3949.038836066" watchObservedRunningTime="2026-11-01 15:23:29.984203672 +0000 UTC m=+3949.042355522" Nov 01 15:23:30.623108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:30.623079 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:30.629095 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:30.629071 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:30.871273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:30.871240 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:30.872085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:30.872064 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:23:31.115533 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:23:31.115492 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:23:31.802951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:31.802920 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:23:32.731976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:32.731936 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:32.901710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:32.901676 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:33.071604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:33.071573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:33.601137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:33.601093 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:36.047390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.047346 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:36.218764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.218729 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:36.594236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.594189 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh"] Nov 01 15:23:36.600279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.600252 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.656862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.656821 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh"] Nov 01 15:23:36.685238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.684227 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jls5x\" (UniqueName: \"kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x\") pod \"payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.685238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.684468 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.700161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.700117 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-04aa47\"" Nov 01 15:23:36.700161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.700155 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-04aa47-dockercfg-s4j94\"" Nov 01 15:23:36.786166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.786127 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.786379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.786276 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jls5x\" (UniqueName: \"kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x\") pod \"payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.786932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.786912 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.843764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.843730 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jls5x\" (UniqueName: \"kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x\") pod \"payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:36.925500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:36.925426 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:37.297278 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:23:37.297244 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380357dc_ed41_4a76_b02c_c3a0641ac252.slice/crio-2df664f665c935f54aa53db54123ce5be0d6df1c3212c10f8499524f0bc61b94 WatchSource:0}: Error finding container 2df664f665c935f54aa53db54123ce5be0d6df1c3212c10f8499524f0bc61b94: Status 404 returned error can't find the container with id 2df664f665c935f54aa53db54123ce5be0d6df1c3212c10f8499524f0bc61b94 Nov 01 15:23:37.301207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:37.301182 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh"] Nov 01 15:23:37.923633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:37.923600 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" event={"ID":"380357dc-ed41-4a76-b02c-c3a0641ac252","Type":"ContainerStarted","Data":"c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51"} Nov 01 15:23:37.923824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:37.923650 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" event={"ID":"380357dc-ed41-4a76-b02c-c3a0641ac252","Type":"ContainerStarted","Data":"2df664f665c935f54aa53db54123ce5be0d6df1c3212c10f8499524f0bc61b94"} Nov 01 15:23:37.923824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:37.923795 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:38.040357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:38.040302 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" podStartSLOduration=2.040286772 podStartE2EDuration="2.040286772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:23:38.036753472 +0000 UTC m=+3957.094905321" watchObservedRunningTime="2026-11-01 15:23:38.040286772 +0000 UTC m=+3957.098438671" Nov 01 15:23:38.533057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:38.533024 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:43.261662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:43.261609 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:47.626341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:47.626302 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:48.934738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:48.934705 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:23:52.016705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:52.016616 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:52.198120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:52.198079 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:52.357875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:52.357795 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:56.598773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:56.598736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:23:57.514806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:23:57.514772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:01.114007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:01.113972 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:01.119102 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:24:01.119074 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:24:04.157414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:04.157385 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:05.503707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:05.503672 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:09.795205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:09.795169 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:09.971153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:09.971106 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:10.137845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:10.137769 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:14.482620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:14.482587 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:18.983237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:18.983178 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:23.516576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:23.516543 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:27.940680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:27.940650 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:31.117876 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:24:31.117841 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:24:32.401143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:32.401112 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:36.770838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:36.770807 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:41.377498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:41.377458 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:45.678669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:45.678635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:45.847284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:45.847248 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:50.277913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:50.277871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:54.529436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.529400 5473 generic.go:356] "Generic (PLEG): container finished" podID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerID="f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63" exitCode=0 Nov 01 15:24:54.529898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.529473 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" event={"ID":"2940993e-3d29-49ca-b4bf-01f2fd2a88f5","Type":"ContainerDied","Data":"f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63"} Nov 01 15:24:54.529898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.529894 5473 scope.go:122] "RemoveContainer" containerID="f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63" Nov 01 15:24:54.531766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.531732 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:24:54.531888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.531832 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:24:54.531888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.531868 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:24:54.628096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:54.628062 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:55.421565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:55.421530 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:24:55.556072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:55.556036 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" event={"ID":"2940993e-3d29-49ca-b4bf-01f2fd2a88f5","Type":"ContainerStarted","Data":"6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7"} Nov 01 15:24:56.181577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:56.181542 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:24:56.183059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:56.183032 5473 patch_prober.go:29] interesting pod/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 01 15:24:56.183189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:56.183089 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 01 15:24:57.015818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:57.015784 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:57.181777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:57.181741 5473 patch_prober.go:29] interesting pod/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 01 15:24:57.181960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:57.181801 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 01 15:24:58.181875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:58.181833 5473 patch_prober.go:29] interesting pod/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 01 15:24:58.182347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:58.181913 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 01 15:24:59.018355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:59.018320 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:24:59.181448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:59.181415 5473 patch_prober.go:29] interesting pod/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 01 15:24:59.181665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:24:59.181479 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 01 15:25:00.181204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:00.181175 5473 patch_prober.go:29] interesting pod/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body= Nov 01 15:25:00.181613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:00.181265 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" Nov 01 15:25:01.110998 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:25:01.110967 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:25:01.186078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:01.186050 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:25:01.186543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:01.186520 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:25:01.187094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:01.187078 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:25:03.434860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:03.434823 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:04.520400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:04.520362 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:07.845201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:07.845166 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:08.016726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:08.016690 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:12.422804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:12.422767 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:13.054517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:13.054484 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr"] Nov 01 15:25:13.104292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:13.104257 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab96954c-6xclr"] Nov 01 15:25:15.084687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:15.084654 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8422c6ce-23b8-4ab1-89dc-10d869fccfb9" path="/var/lib/kubelet/pods/8422c6ce-23b8-4ab1-89dc-10d869fccfb9/volumes" Nov 01 15:25:16.939951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:16.939913 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:21.427083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:21.426999 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:25.834912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:25.834876 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:30.249768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:30.249734 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:31.109804 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:25:31.109760 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:25:32.906343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:32.906310 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:34.790556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:34.790505 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:38.682597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:38.682553 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:39.300566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:39.300527 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:39.871062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:39.871028 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t"] Nov 01 15:25:39.967285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:39.967241 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8"] Nov 01 15:25:40.250924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:40.250804 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:40.412939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:40.412881 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:41.761287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:41.761253 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:43.714363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:43.714328 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:44.184186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:44.184145 5473 scope.go:122] "RemoveContainer" containerID="ef7a53db44365610da6cf7cca0f8c3403fd5798b0ff1448c79a0f2d48128c2bc" Nov 01 15:25:44.560743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:44.560707 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:44.872296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:44.872178 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" podUID="63bbb16e-8b84-4e34-b221-f04ba8dc51a2" containerName="payload-processing" containerID="cri-o://eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797" gracePeriod=25 Nov 01 15:25:44.968191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:44.968147 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" podUID="4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" containerName="payload-pre-processing" containerID="cri-o://986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc" gracePeriod=25 Nov 01 15:25:45.165010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.164982 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:25:45.255623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.255601 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:25:45.314032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.313997 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume\") pod \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " Nov 01 15:25:45.314231 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.314060 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f\" (UniqueName: \"kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f\") pod \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\" (UID: \"63bbb16e-8b84-4e34-b221-f04ba8dc51a2\") " Nov 01 15:25:45.314475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.314443 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume" pod "63bbb16e-8b84-4e34-b221-f04ba8dc51a2" (UID: "63bbb16e-8b84-4e34-b221-f04ba8dc51a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:25:45.314753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.314734 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:25:45.316846 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.316825 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f" pod "63bbb16e-8b84-4e34-b221-f04ba8dc51a2" (UID: "63bbb16e-8b84-4e34-b221-f04ba8dc51a2"). InnerVolumeSpecName "kube-api-access-5vz9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:25:45.415560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.415475 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn\" (UniqueName: \"kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn\") pod \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " Nov 01 15:25:45.415560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.415543 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume\") pod \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\" (UID: \"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0\") " Nov 01 15:25:45.426647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.415949 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume" pod "4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" (UID: "4586c649-8a0a-4542-8fbf-7e7d7dc67fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:25:45.426647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.418285 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn" pod "4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" (UID: "4586c649-8a0a-4542-8fbf-7e7d7dc67fb0"). InnerVolumeSpecName "kube-api-access-bkzbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:25:45.426647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.419413 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzbn\" (UniqueName: \"kubernetes.io/projected/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-kube-api-access-bkzbn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:25:45.426647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.419447 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:25:45.426647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.419464 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vz9f\" (UniqueName: \"kubernetes.io/projected/63bbb16e-8b84-4e34-b221-f04ba8dc51a2-kube-api-access-5vz9f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:25:45.958465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.958426 5473 generic.go:356] "Generic (PLEG): container finished" podID="63bbb16e-8b84-4e34-b221-f04ba8dc51a2" containerID="eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797" exitCode=0 Nov 01 15:25:45.958887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.958479 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" Nov 01 15:25:45.958887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.958503 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" event={"ID":"63bbb16e-8b84-4e34-b221-f04ba8dc51a2","Type":"ContainerDied","Data":"eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797"} Nov 01 15:25:45.958887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.958581 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t" event={"ID":"63bbb16e-8b84-4e34-b221-f04ba8dc51a2","Type":"ContainerDied","Data":"f4fe2c6403c206c0e01805227566a5313ea6eb1d66374a46bc7eb7ebfe3f6e01"} Nov 01 15:25:45.958887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.958605 5473 scope.go:122] "RemoveContainer" containerID="eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797" Nov 01 15:25:45.960474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.960438 5473 generic.go:356] "Generic (PLEG): container finished" podID="4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" containerID="986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc" exitCode=0 Nov 01 15:25:45.960592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.960491 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" event={"ID":"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0","Type":"ContainerDied","Data":"986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc"} Nov 01 15:25:45.960592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.960506 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" Nov 01 15:25:45.960592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.960530 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8" event={"ID":"4586c649-8a0a-4542-8fbf-7e7d7dc67fb0","Type":"ContainerDied","Data":"893eca83e2e615cbde25b1f19a3990476205ef6f47615782ce733a5f6f54aca4"} Nov 01 15:25:45.971926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.971910 5473 scope.go:122] "RemoveContainer" containerID="eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797" Nov 01 15:25:45.972227 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:25:45.972193 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797\": container with ID starting with eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797 not found: ID does not exist" containerID="eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797" Nov 01 15:25:45.972326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.972231 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797"} err="failed to get container status \"eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797\": rpc error: code = NotFound desc = could not find container \"eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797\": container with ID starting with eb42d3b430cbea1eeb1bf569b5bf1d72c2bd1d60e4156b2c3a2f49acb5228797 not found: ID does not exist" Nov 01 15:25:45.972326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.972248 5473 scope.go:122] "RemoveContainer" containerID="986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc" Nov 01 15:25:45.982820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.982799 5473 scope.go:122] "RemoveContainer" containerID="986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc" Nov 01 15:25:45.983089 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:25:45.983069 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc\": container with ID starting with 986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc not found: ID does not exist" containerID="986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc" Nov 01 15:25:45.983189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:45.983093 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc"} err="failed to get container status \"986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc\": rpc error: code = NotFound desc = could not find container \"986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc\": container with ID starting with 986f511214c021061b7abae2c4bfc06b2095b50b5dd26e27a20020684008abfc not found: ID does not exist" Nov 01 15:25:46.124527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.124496 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8"] Nov 01 15:25:46.128208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.128182 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:46.130103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.130081 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-775095-699c46bdcf-5p6d8"] Nov 01 15:25:46.226746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.225605 5473 patch_prober.go:29] interesting pod/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": read tcp 10.134.0.2:57732->10.134.1.50:15021: read: connection reset by peer" start-of-body= Nov 01 15:25:46.226746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.225683 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": read tcp 10.134.0.2:57732->10.134.1.50:15021: read: connection reset by peer" Nov 01 15:25:46.239414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.239388 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t"] Nov 01 15:25:46.245288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.245263 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-775095-5989fc6755-vhs7t"] Nov 01 15:25:46.971544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.971506 5473 generic.go:356] "Generic (PLEG): container finished" podID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerID="fea23a62621cffcb19105fef56ace4319276b434eb71ad1a6ebcdaf0d07c5e87" exitCode=0 Nov 01 15:25:46.971952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.971556 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" event={"ID":"d0c919c2-feaf-4ae3-ba57-48c1d3f41261","Type":"ContainerDied","Data":"fea23a62621cffcb19105fef56ace4319276b434eb71ad1a6ebcdaf0d07c5e87"} Nov 01 15:25:46.971952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.971908 5473 scope.go:122] "RemoveContainer" containerID="fea23a62621cffcb19105fef56ace4319276b434eb71ad1a6ebcdaf0d07c5e87" Nov 01 15:25:46.973572 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.973542 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:25:46.973722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.973625 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:25:46.973722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:46.973675 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:25:47.084164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:47.084137 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" path="/var/lib/kubelet/pods/4586c649-8a0a-4542-8fbf-7e7d7dc67fb0/volumes" Nov 01 15:25:47.084802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:47.084778 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63bbb16e-8b84-4e34-b221-f04ba8dc51a2" path="/var/lib/kubelet/pods/63bbb16e-8b84-4e34-b221-f04ba8dc51a2/volumes" Nov 01 15:25:47.981689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:47.981656 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" event={"ID":"d0c919c2-feaf-4ae3-ba57-48c1d3f41261","Type":"ContainerStarted","Data":"350793c3349b0f0eb385adfa9dc384b12d85c5076dcad22f54fa6bfea4985032"} Nov 01 15:25:48.064057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:48.064022 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:48.623547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:48.623517 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:25:48.623724 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:48.623631 5473 patch_prober.go:29] interesting pod/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 15:25:48.623724 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:48.623678 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 15:25:49.623946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:49.623907 5473 patch_prober.go:29] interesting pod/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 15:25:49.624432 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:49.623963 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 15:25:50.623624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:50.623590 5473 patch_prober.go:29] interesting pod/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 15:25:50.623801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:50.623650 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 15:25:51.623448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:51.623416 5473 patch_prober.go:29] interesting pod/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 15:25:51.623917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:51.623483 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 15:25:52.189164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:52.189133 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:52.354752 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:52.354630 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:52.583445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:52.583410 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:52.629974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:52.629940 5473 patch_prober.go:29] interesting pod/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 01 15:25:52.630626 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:52.630592 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 01 15:25:53.628422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:53.628387 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:25:53.628884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:53.628863 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:25:53.629520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:53.629500 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:25:57.014993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:57.014958 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:25:58.036504 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:25:58.036471 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:01.116189 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:26:01.116152 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:26:01.381917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:01.381832 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:04.817203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:04.817168 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:05.810606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:05.810570 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:10.205400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:10.205364 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:14.601985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:14.601951 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:19.142024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:19.141987 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:19.478284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:19.478178 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:23.511381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:23.511347 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:25.300382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:25.300346 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:25.980635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:25.980600 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn"] Nov 01 15:26:26.351225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:26.351192 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:26.514067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:26.514022 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:28.356993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:28.356959 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:30.836306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:30.836256 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:30.981980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:30.981940 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" podUID="703dd976-3097-496f-a5e5-dec14a920b09" containerName="payload-processing" containerID="cri-o://a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0" gracePeriod=25 Nov 01 15:26:31.115979 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:26:31.115943 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:26:31.262250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.262200 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:26:31.343692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.343662 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume\") pod \"703dd976-3097-496f-a5e5-dec14a920b09\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " Nov 01 15:26:31.343866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.343791 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2\" (UniqueName: \"kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2\") pod \"703dd976-3097-496f-a5e5-dec14a920b09\" (UID: \"703dd976-3097-496f-a5e5-dec14a920b09\") " Nov 01 15:26:31.344052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.344030 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume" pod "703dd976-3097-496f-a5e5-dec14a920b09" (UID: "703dd976-3097-496f-a5e5-dec14a920b09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:26:31.344155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.344135 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/703dd976-3097-496f-a5e5-dec14a920b09-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:26:31.345865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.345843 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2" pod "703dd976-3097-496f-a5e5-dec14a920b09" (UID: "703dd976-3097-496f-a5e5-dec14a920b09"). InnerVolumeSpecName "kube-api-access-nhjl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:26:31.364137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.364066 5473 generic.go:356] "Generic (PLEG): container finished" podID="703dd976-3097-496f-a5e5-dec14a920b09" containerID="a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0" exitCode=0 Nov 01 15:26:31.364137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.364115 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" Nov 01 15:26:31.364321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.364146 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" event={"ID":"703dd976-3097-496f-a5e5-dec14a920b09","Type":"ContainerDied","Data":"a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0"} Nov 01 15:26:31.364321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.364199 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn" event={"ID":"703dd976-3097-496f-a5e5-dec14a920b09","Type":"ContainerDied","Data":"e7569e8583c935fbfa783c5236f8ed3aef2e299b776c8c001621b94426d2eefc"} Nov 01 15:26:31.364321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.364231 5473 scope.go:122] "RemoveContainer" containerID="a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0" Nov 01 15:26:31.375760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.375737 5473 scope.go:122] "RemoveContainer" containerID="a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0" Nov 01 15:26:31.376157 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:26:31.376129 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0\": container with ID starting with a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0 not found: ID does not exist" containerID="a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0" Nov 01 15:26:31.376293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.376168 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0"} err="failed to get container status \"a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0\": rpc error: code = NotFound desc = could not find container \"a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0\": container with ID starting with a333029e90871080b7543e15950dc343484527ebc05ee617c4716ffbd28ca4b0 not found: ID does not exist" Nov 01 15:26:31.445921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.445889 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjl2\" (UniqueName: \"kubernetes.io/projected/703dd976-3097-496f-a5e5-dec14a920b09-kube-api-access-nhjl2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:26:31.490657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.490618 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn"] Nov 01 15:26:31.496433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:31.496408 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc8c64-7c9d4dd9-tpqbn"] Nov 01 15:26:33.082898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:33.082866 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703dd976-3097-496f-a5e5-dec14a920b09" path="/var/lib/kubelet/pods/703dd976-3097-496f-a5e5-dec14a920b09/volumes" Nov 01 15:26:33.178974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:33.178905 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:36.508108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:36.508076 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:37.925807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:37.925757 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:42.338972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:42.338906 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:42.501720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:42.501684 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:42.921058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:42.921023 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl"] Nov 01 15:26:43.027502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:43.027472 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk"] Nov 01 15:26:43.290435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:43.290404 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:43.462108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:43.462075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:44.214931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:44.214895 5473 scope.go:122] "RemoveContainer" containerID="9e1b80c7bba921219d5c6aa8df2ab820b457041a512b955702fa4d06cb88f71b" Nov 01 15:26:47.317071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:47.317034 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:47.921540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:47.921500 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" podUID="657c687b-8dad-45e8-b9dc-fa07caacb5e2" containerName="payload-processing" containerID="cri-o://3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51" gracePeriod=25 Nov 01 15:26:48.028354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.028317 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" podUID="3b798657-7cea-4937-8332-f4c4bec1f369" containerName="payload-pre-processing" containerID="cri-o://2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3" gracePeriod=25 Nov 01 15:26:48.125544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.125502 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:48.196726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.196697 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:26:48.278024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.277993 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume\") pod \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " Nov 01 15:26:48.278231 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.278032 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk\" (UniqueName: \"kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk\") pod \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\" (UID: \"657c687b-8dad-45e8-b9dc-fa07caacb5e2\") " Nov 01 15:26:48.278479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.278450 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume" pod "657c687b-8dad-45e8-b9dc-fa07caacb5e2" (UID: "657c687b-8dad-45e8-b9dc-fa07caacb5e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:26:48.278653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.278636 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657c687b-8dad-45e8-b9dc-fa07caacb5e2-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:26:48.280410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.280385 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk" pod "657c687b-8dad-45e8-b9dc-fa07caacb5e2" (UID: "657c687b-8dad-45e8-b9dc-fa07caacb5e2"). InnerVolumeSpecName "kube-api-access-kmtjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:26:48.302004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.301984 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:26:48.379413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.379375 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume\") pod \"3b798657-7cea-4937-8332-f4c4bec1f369\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " Nov 01 15:26:48.379902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.379420 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf\" (UniqueName: \"kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf\") pod \"3b798657-7cea-4937-8332-f4c4bec1f369\" (UID: \"3b798657-7cea-4937-8332-f4c4bec1f369\") " Nov 01 15:26:48.379902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.379737 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume" pod "3b798657-7cea-4937-8332-f4c4bec1f369" (UID: "3b798657-7cea-4937-8332-f4c4bec1f369"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:26:48.380147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.380125 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b798657-7cea-4937-8332-f4c4bec1f369-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:26:48.380251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.380152 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtjk\" (UniqueName: \"kubernetes.io/projected/657c687b-8dad-45e8-b9dc-fa07caacb5e2-kube-api-access-kmtjk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:26:48.381650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.381630 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf" pod "3b798657-7cea-4937-8332-f4c4bec1f369" (UID: "3b798657-7cea-4937-8332-f4c4bec1f369"). InnerVolumeSpecName "kube-api-access-jm6nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:26:48.481763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.481682 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm6nf\" (UniqueName: \"kubernetes.io/projected/3b798657-7cea-4937-8332-f4c4bec1f369-kube-api-access-jm6nf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:26:48.495027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.494992 5473 generic.go:356] "Generic (PLEG): container finished" podID="657c687b-8dad-45e8-b9dc-fa07caacb5e2" containerID="3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51" exitCode=0 Nov 01 15:26:48.495208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.495041 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" Nov 01 15:26:48.495208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.495077 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" event={"ID":"657c687b-8dad-45e8-b9dc-fa07caacb5e2","Type":"ContainerDied","Data":"3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51"} Nov 01 15:26:48.495208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.495127 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl" event={"ID":"657c687b-8dad-45e8-b9dc-fa07caacb5e2","Type":"ContainerDied","Data":"bd299a18ab1e11145aa2eb8cb8bacb58e25fab10050589d16b99b4d031d1c081"} Nov 01 15:26:48.495208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.495149 5473 scope.go:122] "RemoveContainer" containerID="3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51" Nov 01 15:26:48.496685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.496662 5473 generic.go:356] "Generic (PLEG): container finished" podID="3b798657-7cea-4937-8332-f4c4bec1f369" containerID="2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3" exitCode=0 Nov 01 15:26:48.496783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.496762 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" Nov 01 15:26:48.496783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.496777 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" event={"ID":"3b798657-7cea-4937-8332-f4c4bec1f369","Type":"ContainerDied","Data":"2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3"} Nov 01 15:26:48.496876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.496816 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk" event={"ID":"3b798657-7cea-4937-8332-f4c4bec1f369","Type":"ContainerDied","Data":"be58de0e803e2a21678041520f8d9d0fee14f45d2bf1735e5d636f2a75c287b7"} Nov 01 15:26:48.506639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.506622 5473 scope.go:122] "RemoveContainer" containerID="3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51" Nov 01 15:26:48.506923 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:26:48.506903 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51\": container with ID starting with 3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51 not found: ID does not exist" containerID="3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51" Nov 01 15:26:48.507017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.506929 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51"} err="failed to get container status \"3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51\": rpc error: code = NotFound desc = could not find container \"3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51\": container with ID starting with 3a100d453dbaf137259348f2bb1bca736d2b196be8159152711fbfee50ea3a51 not found: ID does not exist" Nov 01 15:26:48.507017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.506946 5473 scope.go:122] "RemoveContainer" containerID="2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3" Nov 01 15:26:48.518650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.518633 5473 scope.go:122] "RemoveContainer" containerID="2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3" Nov 01 15:26:48.518911 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:26:48.518892 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3\": container with ID starting with 2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3 not found: ID does not exist" containerID="2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3" Nov 01 15:26:48.518967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.518918 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3"} err="failed to get container status \"2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3\": rpc error: code = NotFound desc = could not find container \"2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3\": container with ID starting with 2a0f3e380941ca4d326d251befa5ef43846b6dc6d603a1a935578c905607a8b3 not found: ID does not exist" Nov 01 15:26:48.661108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.661000 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl"] Nov 01 15:26:48.664625 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.664592 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3d5a5b-65ddcfb77-pddnl"] Nov 01 15:26:48.771663 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.771625 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk"] Nov 01 15:26:48.777302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:48.777274 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3d5a5b-5c499c9cc9-m5xbk"] Nov 01 15:26:49.082659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:49.082585 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b798657-7cea-4937-8332-f4c4bec1f369" path="/var/lib/kubelet/pods/3b798657-7cea-4937-8332-f4c4bec1f369/volumes" Nov 01 15:26:49.083084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:49.083063 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657c687b-8dad-45e8-b9dc-fa07caacb5e2" path="/var/lib/kubelet/pods/657c687b-8dad-45e8-b9dc-fa07caacb5e2/volumes" Nov 01 15:26:51.832509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:51.832416 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:26:56.436667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:26:56.436633 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:00.798764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:00.798729 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:01.118465 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:01.118384 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:27:03.516981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.516948 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb"] Nov 01 15:27:03.517640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517624 5473 cpu_manager.go:414] "containerMap: removing container" podUID="703dd976-3097-496f-a5e5-dec14a920b09" containerName="payload-processing" Nov 01 15:27:03.517690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517641 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703dd976-3097-496f-a5e5-dec14a920b09" containerName="payload-processing" Nov 01 15:27:03.517690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517664 5473 cpu_manager.go:414] "containerMap: removing container" podUID="657c687b-8dad-45e8-b9dc-fa07caacb5e2" containerName="payload-processing" Nov 01 15:27:03.517690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517670 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657c687b-8dad-45e8-b9dc-fa07caacb5e2" containerName="payload-processing" Nov 01 15:27:03.517690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517681 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3b798657-7cea-4937-8332-f4c4bec1f369" containerName="payload-pre-processing" Nov 01 15:27:03.517690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517687 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b798657-7cea-4937-8332-f4c4bec1f369" containerName="payload-pre-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517697 5473 cpu_manager.go:414] "containerMap: removing container" podUID="63bbb16e-8b84-4e34-b221-f04ba8dc51a2" containerName="payload-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517702 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63bbb16e-8b84-4e34-b221-f04ba8dc51a2" containerName="payload-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517725 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" containerName="payload-pre-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517730 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" containerName="payload-pre-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517821 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="657c687b-8dad-45e8-b9dc-fa07caacb5e2" containerName="payload-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517831 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b798657-7cea-4937-8332-f4c4bec1f369" containerName="payload-pre-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517837 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="63bbb16e-8b84-4e34-b221-f04ba8dc51a2" containerName="payload-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517847 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4586c649-8a0a-4542-8fbf-7e7d7dc67fb0" containerName="payload-pre-processing" Nov 01 15:27:03.517857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.517857 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="703dd976-3097-496f-a5e5-dec14a920b09" containerName="payload-processing" Nov 01 15:27:03.522875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.522858 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.578041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.578012 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb"] Nov 01 15:27:03.623761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.623733 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:27:03.623928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.623806 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:27:03.687881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.687848 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.688084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.687909 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.688168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.688143 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xqx\" (UniqueName: \"kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.789548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.789459 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.789548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.789532 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.789796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.789689 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xqx\" (UniqueName: \"kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.790270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.790246 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.791848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.791818 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:03.846580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:03.846555 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xqx\" (UniqueName: \"kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx\") pod \"maas-api-revoke-keys-e2e-inf-a-775095-gtwlb\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:04.137275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:04.137160 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:04.506006 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:27:04.505976 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0f8ed6_a18c_4cc5_b863_990c95f3ef51.slice/crio-9f912f671e6a36169f98dbd36c69bb9075b50ee9e4ba4838b8a07bd8f7bc20d6 WatchSource:0}: Error finding container 9f912f671e6a36169f98dbd36c69bb9075b50ee9e4ba4838b8a07bd8f7bc20d6: Status 404 returned error can't find the container with id 9f912f671e6a36169f98dbd36c69bb9075b50ee9e4ba4838b8a07bd8f7bc20d6 Nov 01 15:27:04.509509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:04.509484 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb"] Nov 01 15:27:04.620891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:04.620859 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" event={"ID":"af0f8ed6-a18c-4cc5-b863-990c95f3ef51","Type":"ContainerStarted","Data":"9f912f671e6a36169f98dbd36c69bb9075b50ee9e4ba4838b8a07bd8f7bc20d6"} Nov 01 15:27:05.137026 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:05.136940 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:05.370085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:05.370048 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:05.629819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:05.629783 5473 generic.go:356] "Generic (PLEG): container finished" podID="af0f8ed6-a18c-4cc5-b863-990c95f3ef51" containerID="bde400045c3a78b77710611256517a90593a749dea7295bfe3a727c28d20f4be" exitCode=0 Nov 01 15:27:05.630280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:05.629867 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" event={"ID":"af0f8ed6-a18c-4cc5-b863-990c95f3ef51","Type":"ContainerDied","Data":"bde400045c3a78b77710611256517a90593a749dea7295bfe3a727c28d20f4be"} Nov 01 15:27:06.798817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.798788 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:06.824723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.824693 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca\") pod \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " Nov 01 15:27:06.824870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.824750 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca\") pod \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " Nov 01 15:27:06.824870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.824819 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx\" (UniqueName: \"kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx\") pod \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\" (UID: \"af0f8ed6-a18c-4cc5-b863-990c95f3ef51\") " Nov 01 15:27:06.825110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.825084 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca" pod "af0f8ed6-a18c-4cc5-b863-990c95f3ef51" (UID: "af0f8ed6-a18c-4cc5-b863-990c95f3ef51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:27:06.825490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.825468 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:06.826801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.826780 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca" pod "af0f8ed6-a18c-4cc5-b863-990c95f3ef51" (UID: "af0f8ed6-a18c-4cc5-b863-990c95f3ef51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:27:06.826910 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.826887 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx" pod "af0f8ed6-a18c-4cc5-b863-990c95f3ef51" (UID: "af0f8ed6-a18c-4cc5-b863-990c95f3ef51"). InnerVolumeSpecName "kube-api-access-v8xqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:06.927229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.927130 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:06.927229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:06.927160 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xqx\" (UniqueName: \"kubernetes.io/projected/af0f8ed6-a18c-4cc5-b863-990c95f3ef51-kube-api-access-v8xqx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:07.645390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:07.645353 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" event={"ID":"af0f8ed6-a18c-4cc5-b863-990c95f3ef51","Type":"ContainerDied","Data":"9f912f671e6a36169f98dbd36c69bb9075b50ee9e4ba4838b8a07bd8f7bc20d6"} Nov 01 15:27:07.645581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:07.645403 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f912f671e6a36169f98dbd36c69bb9075b50ee9e4ba4838b8a07bd8f7bc20d6" Nov 01 15:27:07.645581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:07.645366 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb" Nov 01 15:27:09.824075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:09.824042 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:14.051114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:14.051070 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs"] Nov 01 15:27:14.099920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:14.099887 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0aa222-r8lqs"] Nov 01 15:27:14.268482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:14.268448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:15.082771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:15.082732 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e2d2507-57d8-43f8-8837-317145332ec3" path="/var/lib/kubelet/pods/3e2d2507-57d8-43f8-8837-317145332ec3/volumes" Nov 01 15:27:16.778501 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.778470 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l"] Nov 01 15:27:16.779118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.779104 5473 cpu_manager.go:414] "containerMap: removing container" podUID="af0f8ed6-a18c-4cc5-b863-990c95f3ef51" containerName="revoke-keys" Nov 01 15:27:16.779165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.779119 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0f8ed6-a18c-4cc5-b863-990c95f3ef51" containerName="revoke-keys" Nov 01 15:27:16.779287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.779277 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0f8ed6-a18c-4cc5-b863-990c95f3ef51" containerName="revoke-keys" Nov 01 15:27:16.783009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.782989 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.839501 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.839469 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l"] Nov 01 15:27:16.847961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.847928 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjgk\" (UniqueName: \"kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.848153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.848043 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.848333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.848307 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.884650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.884628 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:27:16.884650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.884648 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:27:16.949593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.949569 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.949728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.949666 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjgk\" (UniqueName: \"kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.949728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.949708 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.950405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.950384 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:16.951921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:16.951904 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:17.006269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:17.006245 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjgk\" (UniqueName: \"kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk\") pod \"maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:17.097485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:17.097406 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:17.471274 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:27:17.471204 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda23cdbf_7d67_4d92_9608_e4a2e04cfe4a.slice/crio-04e72f196a89ca4de75ce4d6f8d23adca85407ae0f485af1367e0915b0492d41 WatchSource:0}: Error finding container 04e72f196a89ca4de75ce4d6f8d23adca85407ae0f485af1367e0915b0492d41: Status 404 returned error can't find the container with id 04e72f196a89ca4de75ce4d6f8d23adca85407ae0f485af1367e0915b0492d41 Nov 01 15:27:17.475995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:17.475967 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l"] Nov 01 15:27:17.718419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:17.718336 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" event={"ID":"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a","Type":"ContainerStarted","Data":"04e72f196a89ca4de75ce4d6f8d23adca85407ae0f485af1367e0915b0492d41"} Nov 01 15:27:18.727671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:18.727590 5473 generic.go:356] "Generic (PLEG): container finished" podID="da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" containerID="1dfd350677a22d5a2d216604642e12dc6bdd99be44f41910ae4931ef6d41e1b7" exitCode=0 Nov 01 15:27:18.728046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:18.727669 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" event={"ID":"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a","Type":"ContainerDied","Data":"1dfd350677a22d5a2d216604642e12dc6bdd99be44f41910ae4931ef6d41e1b7"} Nov 01 15:27:18.845776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:18.845745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:19.909456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.909430 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:19.981111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.981077 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:19.990419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.990359 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca\") pod \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " Nov 01 15:27:19.990529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.990505 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca\") pod \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " Nov 01 15:27:19.990529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.990526 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk\" (UniqueName: \"kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk\") pod \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\" (UID: \"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a\") " Nov 01 15:27:19.990849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.990821 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca" pod "da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" (UID: "da23cdbf-7d67-4d92-9608-e4a2e04cfe4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:27:19.991004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.990989 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:19.992420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.992398 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca" pod "da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" (UID: "da23cdbf-7d67-4d92-9608-e4a2e04cfe4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:27:19.992604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:19.992586 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk" pod "da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" (UID: "da23cdbf-7d67-4d92-9608-e4a2e04cfe4a"). InnerVolumeSpecName "kube-api-access-gdjgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:20.092704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:20.092667 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:20.092704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:20.092704 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjgk\" (UniqueName: \"kubernetes.io/projected/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a-kube-api-access-gdjgk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:20.743311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:20.743268 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" event={"ID":"da23cdbf-7d67-4d92-9608-e4a2e04cfe4a","Type":"ContainerDied","Data":"04e72f196a89ca4de75ce4d6f8d23adca85407ae0f485af1367e0915b0492d41"} Nov 01 15:27:20.743502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:20.743326 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04e72f196a89ca4de75ce4d6f8d23adca85407ae0f485af1367e0915b0492d41" Nov 01 15:27:20.743502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:20.743287 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l" Nov 01 15:27:23.351411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:23.351353 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:24.090612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.090578 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh"] Nov 01 15:27:24.091264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.091249 5473 cpu_manager.go:414] "containerMap: removing container" podUID="da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" containerName="revoke-keys" Nov 01 15:27:24.091325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.091264 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" containerName="revoke-keys" Nov 01 15:27:24.091367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.091358 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" containerName="revoke-keys" Nov 01 15:27:24.095021 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.095006 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.151979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.151949 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh"] Nov 01 15:27:24.197357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.197313 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:27:24.197357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.197337 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:27:24.247336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.247305 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.247503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.247377 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.247556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.247524 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krzzl\" (UniqueName: \"kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.348666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.348591 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.348815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.348732 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.348815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.348767 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krzzl\" (UniqueName: \"kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.349404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.349378 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.350978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.350956 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.406015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.405987 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krzzl\" (UniqueName: \"kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl\") pod \"maas-api-revoke-sub-7fe9cfcd-rckqh\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.408986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.408964 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:24.778161 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:27:24.778134 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c97256e_a9cf_4bc7_8d45_f199bbc97566.slice/crio-fcd017f3eb8369680a3843d31cd11756f2613715e1b52ce8047d4aefdb80a04d WatchSource:0}: Error finding container fcd017f3eb8369680a3843d31cd11756f2613715e1b52ce8047d4aefdb80a04d: Status 404 returned error can't find the container with id fcd017f3eb8369680a3843d31cd11756f2613715e1b52ce8047d4aefdb80a04d Nov 01 15:27:24.782469 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:24.782446 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh"] Nov 01 15:27:25.782525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:25.782491 5473 generic.go:356] "Generic (PLEG): container finished" podID="9c97256e-a9cf-4bc7-8d45-f199bbc97566" containerID="7913ceac2900d5d7509236e0c07e7c7caeaa7fd215daa5debf47ddce655c0d6c" exitCode=0 Nov 01 15:27:25.782915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:25.782532 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" event={"ID":"9c97256e-a9cf-4bc7-8d45-f199bbc97566","Type":"ContainerDied","Data":"7913ceac2900d5d7509236e0c07e7c7caeaa7fd215daa5debf47ddce655c0d6c"} Nov 01 15:27:25.782915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:25.782578 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" event={"ID":"9c97256e-a9cf-4bc7-8d45-f199bbc97566","Type":"ContainerStarted","Data":"fcd017f3eb8369680a3843d31cd11756f2613715e1b52ce8047d4aefdb80a04d"} Nov 01 15:27:26.947679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:26.947653 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:27.083566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.083491 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca\") pod \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " Nov 01 15:27:27.083566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.083532 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl\" (UniqueName: \"kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl\") pod \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " Nov 01 15:27:27.083810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.083617 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca\") pod \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\" (UID: \"9c97256e-a9cf-4bc7-8d45-f199bbc97566\") " Nov 01 15:27:27.083918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.083893 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca" pod "9c97256e-a9cf-4bc7-8d45-f199bbc97566" (UID: "9c97256e-a9cf-4bc7-8d45-f199bbc97566"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:27:27.084456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.084429 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:27.085870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.085834 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca" pod "9c97256e-a9cf-4bc7-8d45-f199bbc97566" (UID: "9c97256e-a9cf-4bc7-8d45-f199bbc97566"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:27:27.085995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.085928 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl" pod "9c97256e-a9cf-4bc7-8d45-f199bbc97566" (UID: "9c97256e-a9cf-4bc7-8d45-f199bbc97566"). InnerVolumeSpecName "kube-api-access-krzzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:27.185493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.185459 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c97256e-a9cf-4bc7-8d45-f199bbc97566-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:27.185493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.185490 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krzzl\" (UniqueName: \"kubernetes.io/projected/9c97256e-a9cf-4bc7-8d45-f199bbc97566-kube-api-access-krzzl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:27.709332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.709294 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:27.800277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.800243 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" event={"ID":"9c97256e-a9cf-4bc7-8d45-f199bbc97566","Type":"ContainerDied","Data":"fcd017f3eb8369680a3843d31cd11756f2613715e1b52ce8047d4aefdb80a04d"} Nov 01 15:27:27.800277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.800263 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh" Nov 01 15:27:27.800528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:27.800292 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcd017f3eb8369680a3843d31cd11756f2613715e1b52ce8047d4aefdb80a04d" Nov 01 15:27:31.109692 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:31.109649 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:27:32.156535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:32.156496 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:36.424598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:36.424563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:36.590064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:36.590019 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:38.800098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:38.800065 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:38.986062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:38.986026 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:40.458164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:40.458135 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:42.139643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.139610 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:42.302082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.302051 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:42.797548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.797519 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww"] Nov 01 15:27:42.798241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.798208 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9c97256e-a9cf-4bc7-8d45-f199bbc97566" containerName="revoke-keys" Nov 01 15:27:42.798308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.798241 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c97256e-a9cf-4bc7-8d45-f199bbc97566" containerName="revoke-keys" Nov 01 15:27:42.798374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.798364 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c97256e-a9cf-4bc7-8d45-f199bbc97566" containerName="revoke-keys" Nov 01 15:27:42.802409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.802393 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:42.856951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.856920 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww"] Nov 01 15:27:42.902893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.902863 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 15:27:42.903059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.902990 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 15:27:42.903059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.902994 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 15:27:42.904781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.904762 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-99e3d9db4e44\"" Nov 01 15:27:42.904781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.904776 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 15:27:42.904928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.904790 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-t6xwv\"" Nov 01 15:27:42.923491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.923462 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:42.923598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.923528 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:42.923673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.923659 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:42.923722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:42.923710 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvrw\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.024800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.024770 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvrw\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.024966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.024906 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.025027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.024968 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.025134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.025110 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.025773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.025750 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.025860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.025841 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.027716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.027693 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.081825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.081753 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvrw\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw\") pod \"payload-processing-external-model-6ff6c87fcc-47vww\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.116523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.116429 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.588028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.587994 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww"] Nov 01 15:27:43.925039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.924948 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" event={"ID":"43a9ee0a-b349-4848-809f-5d59d4a6c309","Type":"ContainerStarted","Data":"d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071"} Nov 01 15:27:43.925039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.925004 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:43.925039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:43.925023 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" event={"ID":"43a9ee0a-b349-4848-809f-5d59d4a6c309","Type":"ContainerStarted","Data":"a7d9d1cd02828add8a31ec81a1c0f937bcd250af009b0b67ccfc8bddf3192bf5"} Nov 01 15:27:44.038625 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:44.038568 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" podStartSLOduration=2.038555721 podStartE2EDuration="2.038555721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:27:44.037963903 +0000 UTC m=+4203.096115754" watchObservedRunningTime="2026-11-01 15:27:44.038555721 +0000 UTC m=+4203.096707570" Nov 01 15:27:44.547468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:44.547442 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:27:44.560565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:44.560543 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:27:44.927085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:44.927044 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:45.088097 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:45.088059 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:45.088097 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:45.088088 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:45.088348 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:45.088143 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials podName:43a9ee0a-b349-4848-809f-5d59d4a6c309 nodeName:}" failed. No retries permitted until 2026-11-01 15:27:45.588126399 +0000 UTC m=+4204.646278231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-47vww" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:45.090595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:45.090556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:45.595820 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:45.595785 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:45.595820 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:45.595809 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:45.596048 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:45.595866 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials podName:43a9ee0a-b349-4848-809f-5d59d4a6c309 nodeName:}" failed. No retries permitted until 2026-11-01 15:27:46.595850949 +0000 UTC m=+4205.654002789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-47vww" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:45.968224 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:45.968127 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww"] Nov 01 15:27:46.135435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:46.135404 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:46.610949 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:46.610914 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:46.610949 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:46.610941 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:46.611172 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:46.611005 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials podName:43a9ee0a-b349-4848-809f-5d59d4a6c309 nodeName:}" failed. No retries permitted until 2026-11-01 15:27:48.61099014 +0000 UTC m=+4207.669141987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-47vww" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:46.675811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:46.675779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:47.885332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:47.885284 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:48.051192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:48.051158 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:48.641598 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:48.641565 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:48.641598 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:48.641590 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:48.641817 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:48.641639 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials podName:43a9ee0a-b349-4848-809f-5d59d4a6c309 nodeName:}" failed. No retries permitted until 2026-11-01 15:27:52.641627075 +0000 UTC m=+4211.699778921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-47vww" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:49.351872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:49.351837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:49.950313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:49.950276 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58"] Nov 01 15:27:49.950495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:49.950479 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" containerID="cri-o://6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3" gracePeriod=30 Nov 01 15:27:49.972233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:49.972194 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:50.691777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:50.691740 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:50.968431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:50.968347 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" podUID="43a9ee0a-b349-4848-809f-5d59d4a6c309" containerName="payload-processing" containerID="cri-o://d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071" gracePeriod=25 Nov 01 15:27:51.017513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.017480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:51.168580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.168549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:51.268973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.268944 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:51.460113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.456550 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume\") pod \"43a9ee0a-b349-4848-809f-5d59d4a6c309\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " Nov 01 15:27:51.460113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.456608 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw\") pod \"43a9ee0a-b349-4848-809f-5d59d4a6c309\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " Nov 01 15:27:51.460113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.456660 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials\") pod \"43a9ee0a-b349-4848-809f-5d59d4a6c309\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " Nov 01 15:27:51.460113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.456784 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume\") pod \"43a9ee0a-b349-4848-809f-5d59d4a6c309\" (UID: \"43a9ee0a-b349-4848-809f-5d59d4a6c309\") " Nov 01 15:27:51.460113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.457583 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume" pod "43a9ee0a-b349-4848-809f-5d59d4a6c309" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:27:51.460113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.457955 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume" pod "43a9ee0a-b349-4848-809f-5d59d4a6c309" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:27:51.464777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.464532 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw" pod "43a9ee0a-b349-4848-809f-5d59d4a6c309" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309"). InnerVolumeSpecName "kube-api-access-gpvrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:51.471754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.471723 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials" pod "43a9ee0a-b349-4848-809f-5d59d4a6c309" (UID: "43a9ee0a-b349-4848-809f-5d59d4a6c309"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:51.518868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.518843 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:51.559089 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.559051 5473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-routing-overlay-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:51.559369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.559348 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvrw\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-kube-api-access-gpvrw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:51.559483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.559471 5473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/43a9ee0a-b349-4848-809f-5d59d4a6c309-provider-credentials\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:51.559581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.559569 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43a9ee0a-b349-4848-809f-5d59d4a6c309-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:51.990991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.990904 5473 generic.go:356] "Generic (PLEG): container finished" podID="43a9ee0a-b349-4848-809f-5d59d4a6c309" containerID="d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071" exitCode=0 Nov 01 15:27:51.990991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.990947 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" Nov 01 15:27:51.991603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.990982 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" event={"ID":"43a9ee0a-b349-4848-809f-5d59d4a6c309","Type":"ContainerDied","Data":"d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071"} Nov 01 15:27:51.991603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.991046 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww" event={"ID":"43a9ee0a-b349-4848-809f-5d59d4a6c309","Type":"ContainerDied","Data":"a7d9d1cd02828add8a31ec81a1c0f937bcd250af009b0b67ccfc8bddf3192bf5"} Nov 01 15:27:51.991603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:51.991077 5473 scope.go:122] "RemoveContainer" containerID="d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071" Nov 01 15:27:52.002565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:52.002537 5473 scope.go:122] "RemoveContainer" containerID="d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071" Nov 01 15:27:52.002854 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:52.002836 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071\": container with ID starting with d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071 not found: ID does not exist" containerID="d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071" Nov 01 15:27:52.002910 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:52.002865 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071"} err="failed to get container status \"d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071\": rpc error: code = NotFound desc = could not find container \"d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071\": container with ID starting with d32689ea1252afe7392715bf71be79d8dd948b32b8374990902934a5940c0071 not found: ID does not exist" Nov 01 15:27:52.157667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:52.157633 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww"] Nov 01 15:27:52.207403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:52.207369 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-47vww"] Nov 01 15:27:52.810933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:52.810904 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:53.103128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:53.098509 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a9ee0a-b349-4848-809f-5d59d4a6c309" path="/var/lib/kubelet/pods/43a9ee0a-b349-4848-809f-5d59d4a6c309/volumes" Nov 01 15:27:53.974235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:53.974173 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7"] Nov 01 15:27:53.975346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:53.975320 5473 cpu_manager.go:414] "containerMap: removing container" podUID="43a9ee0a-b349-4848-809f-5d59d4a6c309" containerName="payload-processing" Nov 01 15:27:53.975346 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:53.975347 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a9ee0a-b349-4848-809f-5d59d4a6c309" containerName="payload-processing" Nov 01 15:27:53.975573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:53.975557 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a9ee0a-b349-4848-809f-5d59d4a6c309" containerName="payload-processing" Nov 01 15:27:53.981097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:53.981074 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.039241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.037730 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7"] Nov 01 15:27:54.054680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.054651 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 15:27:54.055016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.054998 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 15:27:54.071239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.055196 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 15:27:54.081613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.081582 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 15:27:54.081870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.081849 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-99e3d9db4e44\"" Nov 01 15:27:54.082133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.082114 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-zls5w\"" Nov 01 15:27:54.166715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.166117 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.166715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.166352 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlqwp\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.166715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.166481 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.166715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.166541 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.275234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.272938 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.275234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.273165 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlqwp\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.275234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.273310 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.275234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.273348 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.311559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.308999 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.329197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.329160 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.341265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.341234 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.392355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.392318 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlqwp\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp\") pod \"payload-processing-external-model-6ff6c87fcc-vtgh7\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:54.601290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:54.600501 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:55.343251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:55.332338 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 01 15:27:55.718520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:55.718479 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 15:27:55.771532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:55.771491 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Nov 01 15:27:55.789834 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:55.789792 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:27:56.136191 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:56.136154 5473 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.056s" Nov 01 15:27:56.136561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.136533 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7"] Nov 01 15:27:56.175566 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:27:56.175525 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc767e474_9ab0_4e34_a6ed_92656dd6f156.slice/crio-17583539f180b62c3fac3c2a74799d0aa1033bd373bf30b17a66112e18bedc65 WatchSource:0}: Error finding container 17583539f180b62c3fac3c2a74799d0aa1033bd373bf30b17a66112e18bedc65: Status 404 returned error can't find the container with id 17583539f180b62c3fac3c2a74799d0aa1033bd373bf30b17a66112e18bedc65 Nov 01 15:27:56.414613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.414580 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:27:56.516397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516370 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516427 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516491 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516520 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516662 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516703 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:27:56.516850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516727 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:27:56.516850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516745 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516794 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.516850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516837 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.517084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516880 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert\") pod \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\" (UID: \"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803\") " Nov 01 15:27:56.517084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.516971 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:27:56.517210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517108 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:27:56.517399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517339 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:27:56.517609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517593 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.517692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517615 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.517692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517629 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.517692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517642 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.517692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.517656 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.518858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.518838 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:27:56.518978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.518939 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:27:56.519288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.519270 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:56.519366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.519296 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg" pod "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" (UID: "4fb94eef-c4a0-4f62-aa95-3b1b7f69f803"). InnerVolumeSpecName "kube-api-access-fh4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:27:56.551468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.551446 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:56.618915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.618832 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.618915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.618870 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.618915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.618889 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.618915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.618904 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4pg\" (UniqueName: \"kubernetes.io/projected/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803-kube-api-access-fh4pg\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:27:56.826018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.825970 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" event={"ID":"c767e474-9ab0-4e34-a6ed-92656dd6f156","Type":"ContainerStarted","Data":"315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665"} Nov 01 15:27:56.826205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.826043 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:27:56.826205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.826063 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" event={"ID":"c767e474-9ab0-4e34-a6ed-92656dd6f156","Type":"ContainerStarted","Data":"17583539f180b62c3fac3c2a74799d0aa1033bd373bf30b17a66112e18bedc65"} Nov 01 15:27:56.827761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.827734 5473 generic.go:356] "Generic (PLEG): container finished" podID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerID="6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3" exitCode=0 Nov 01 15:27:56.827874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.827770 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" event={"ID":"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803","Type":"ContainerDied","Data":"6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3"} Nov 01 15:27:56.827874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.827782 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" Nov 01 15:27:56.827874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.827813 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58" event={"ID":"4fb94eef-c4a0-4f62-aa95-3b1b7f69f803","Type":"ContainerDied","Data":"c7df585e0819f85f7d1d8262ff18c35f131de959333516dcc405281b8bcbba82"} Nov 01 15:27:56.827874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.827842 5473 scope.go:122] "RemoveContainer" containerID="6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3" Nov 01 15:27:56.839112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.839095 5473 scope.go:122] "RemoveContainer" containerID="4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70" Nov 01 15:27:56.848769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.848747 5473 scope.go:122] "RemoveContainer" containerID="6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3" Nov 01 15:27:56.849048 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:56.849023 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3\": container with ID starting with 6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3 not found: ID does not exist" containerID="6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3" Nov 01 15:27:56.849141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.849055 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3"} err="failed to get container status \"6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3\": rpc error: code = NotFound desc = could not find container \"6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3\": container with ID starting with 6d913c753345f1051260ca6d695394c70648153c5c0bd9dadec3a9666672cfc3 not found: ID does not exist" Nov 01 15:27:56.849141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.849071 5473 scope.go:122] "RemoveContainer" containerID="4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70" Nov 01 15:27:56.849383 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:56.849367 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70\": container with ID starting with 4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70 not found: ID does not exist" containerID="4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70" Nov 01 15:27:56.849437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.849387 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70"} err="failed to get container status \"4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70\": rpc error: code = NotFound desc = could not find container \"4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70\": container with ID starting with 4993e37273c837c6494d9798a074ade6bbea2aed2b7c4480e87c6075c55e9d70 not found: ID does not exist" Nov 01 15:27:56.942949 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:56.942847 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" podStartSLOduration=3.942834289 podStartE2EDuration="3.942834289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:27:56.9420833 +0000 UTC m=+4216.000235151" watchObservedRunningTime="2026-11-01 15:27:56.942834289 +0000 UTC m=+4216.000986162" Nov 01 15:27:57.058070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.058034 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58"] Nov 01 15:27:57.062062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.062025 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fc8c64-openshift-default-5f4dfd647f-4kh58"] Nov 01 15:27:57.083660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.083621 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" path="/var/lib/kubelet/pods/4fb94eef-c4a0-4f62-aa95-3b1b7f69f803/volumes" Nov 01 15:27:57.256535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.256451 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:57.487506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.487472 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:57.642514 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.642480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:57.928247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:57.928155 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7"] Nov 01 15:27:57.946778 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:57.946747 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:57.946778 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:57.946772 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:57.947012 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:57.946834 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials podName:c767e474-9ab0-4e34-a6ed-92656dd6f156 nodeName:}" failed. No retries permitted until 2026-11-01 15:27:58.446816364 +0000 UTC m=+4217.504968206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-vtgh7" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:58.099732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:58.099703 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:58.453573 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:58.453522 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:58.453573 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:58.453568 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:58.453814 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:58.453628 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials podName:c767e474-9ab0-4e34-a6ed-92656dd6f156 nodeName:}" failed. No retries permitted until 2026-11-01 15:27:59.453612568 +0000 UTC m=+4218.511764419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-vtgh7" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:59.329428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.329396 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:59.466159 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:59.466125 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:59.466159 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:59.466152 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:59.466426 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:27:59.466205 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials podName:c767e474-9ab0-4e34-a6ed-92656dd6f156 nodeName:}" failed. No retries permitted until 2026-11-01 15:28:01.466191424 +0000 UTC m=+4220.524343266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-vtgh7" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:27:59.755372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.755275 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm"] Nov 01 15:27:59.756056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.756034 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" Nov 01 15:27:59.756056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.756056 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" Nov 01 15:27:59.756301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.756287 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" Nov 01 15:27:59.756380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.756307 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" Nov 01 15:27:59.761776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.761755 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.816567 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96165a-openshift-default-dockercfg-srcxq\"" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.872616 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.872925 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873025 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvgz\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873111 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873182 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873235 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873294 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873326 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.873370 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.899935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.876113 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:27:59.938033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.937979 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm"] Nov 01 15:27:59.974664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974626 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.974876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974694 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.974876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974719 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.974876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974756 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.974876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974773 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.974876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974799 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.974876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974824 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.975454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974900 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.975454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.974959 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvgz\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.975454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.975193 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.975634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.975517 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.975718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.975698 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.975891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.975868 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.976272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.976248 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.980317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.980260 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:27:59.980679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:27:59.980658 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:00.034282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.034249 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:00.034451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.034283 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvgz\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz\") pod \"e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:00.075898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.075866 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:00.276680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.276647 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:00.496496 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:28:00.496456 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f9ff98_2bbc_4126_b236_9bcc598d02d6.slice/crio-f8233e04f508a25490c2e4f8c88938da3c80f514814c372d35b7dcee2d8f3c5a WatchSource:0}: Error finding container f8233e04f508a25490c2e4f8c88938da3c80f514814c372d35b7dcee2d8f3c5a: Status 404 returned error can't find the container with id f8233e04f508a25490c2e4f8c88938da3c80f514814c372d35b7dcee2d8f3c5a Nov 01 15:28:00.499323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.499292 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:28:00.499415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.499357 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:28:00.499415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.499391 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:28:00.500780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.500755 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm"] Nov 01 15:28:00.626926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.626887 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n"] Nov 01 15:28:00.881051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.881008 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" event={"ID":"88f9ff98-2bbc-4126-b236-9bcc598d02d6","Type":"ContainerStarted","Data":"2358c93315e86b82a3574abc11ed577a7be55c205df05bc2cb7e36b5580545a1"} Nov 01 15:28:00.881276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.881079 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" event={"ID":"88f9ff98-2bbc-4126-b236-9bcc598d02d6","Type":"ContainerStarted","Data":"f8233e04f508a25490c2e4f8c88938da3c80f514814c372d35b7dcee2d8f3c5a"} Nov 01 15:28:00.905470 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:00.905438 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:01.005338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:01.000368 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podStartSLOduration=2.00034953 podStartE2EDuration="2.00034953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:28:00.998425908 +0000 UTC m=+4220.056577785" watchObservedRunningTime="2026-11-01 15:28:01.00034953 +0000 UTC m=+4220.058501401" Nov 01 15:28:01.068434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:01.068398 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:01.084742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:01.084716 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:01.084900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:01.084803 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:01.108820 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:01.108780 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:28:01.495826 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:01.495797 5473 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-99e3d9db4e44: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:28:01.495826 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:01.495821 5473 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7: secret "agc-cred-99e3d9db4e44" not found Nov 01 15:28:01.496055 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:01.495887 5473 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials podName:c767e474-9ab0-4e34-a6ed-92656dd6f156 nodeName:}" failed. No retries permitted until 2026-11-01 15:28:05.49587261 +0000 UTC m=+4224.554024454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials") pod "payload-processing-external-model-6ff6c87fcc-vtgh7" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156") : secret "agc-cred-99e3d9db4e44" not found Nov 01 15:28:01.892627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:01.890382 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:01.892627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:01.892074 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:28:02.746036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:02.746007 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:03.858097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:03.858060 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" podUID="c767e474-9ab0-4e34-a6ed-92656dd6f156" containerName="payload-processing" containerID="cri-o://315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665" gracePeriod=25 Nov 01 15:28:03.892347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:03.892321 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:04.077267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.077232 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:04.125578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.125517 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:28:04.257894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.257861 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp\") pod \"c767e474-9ab0-4e34-a6ed-92656dd6f156\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " Nov 01 15:28:04.258101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.257951 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials\") pod \"c767e474-9ab0-4e34-a6ed-92656dd6f156\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " Nov 01 15:28:04.258187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.258138 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume\") pod \"c767e474-9ab0-4e34-a6ed-92656dd6f156\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " Nov 01 15:28:04.258187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.258170 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume\") pod \"c767e474-9ab0-4e34-a6ed-92656dd6f156\" (UID: \"c767e474-9ab0-4e34-a6ed-92656dd6f156\") " Nov 01 15:28:04.258596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.258564 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume" pod "c767e474-9ab0-4e34-a6ed-92656dd6f156" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:28:04.258727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.258707 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume" pod "c767e474-9ab0-4e34-a6ed-92656dd6f156" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:28:04.258864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.258848 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:04.258935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.258865 5473 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c767e474-9ab0-4e34-a6ed-92656dd6f156-routing-overlay-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:04.260230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.260194 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp" pod "c767e474-9ab0-4e34-a6ed-92656dd6f156" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156"). InnerVolumeSpecName "kube-api-access-hlqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:28:04.260435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.260415 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials" pod "c767e474-9ab0-4e34-a6ed-92656dd6f156" (UID: "c767e474-9ab0-4e34-a6ed-92656dd6f156"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:28:04.359749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.359712 5473 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-provider-credentials\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:04.359749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.359749 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlqwp\" (UniqueName: \"kubernetes.io/projected/c767e474-9ab0-4e34-a6ed-92656dd6f156-kube-api-access-hlqwp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:04.915691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.915652 5473 generic.go:356] "Generic (PLEG): container finished" podID="c767e474-9ab0-4e34-a6ed-92656dd6f156" containerID="315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665" exitCode=0 Nov 01 15:28:04.916130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.915714 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" Nov 01 15:28:04.916130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.915725 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" event={"ID":"c767e474-9ab0-4e34-a6ed-92656dd6f156","Type":"ContainerDied","Data":"315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665"} Nov 01 15:28:04.916130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.915811 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7" event={"ID":"c767e474-9ab0-4e34-a6ed-92656dd6f156","Type":"ContainerDied","Data":"17583539f180b62c3fac3c2a74799d0aa1033bd373bf30b17a66112e18bedc65"} Nov 01 15:28:04.916130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.915844 5473 scope.go:122] "RemoveContainer" containerID="315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665" Nov 01 15:28:04.930720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.930698 5473 scope.go:122] "RemoveContainer" containerID="315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665" Nov 01 15:28:04.931048 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:04.931025 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665\": container with ID starting with 315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665 not found: ID does not exist" containerID="315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665" Nov 01 15:28:04.931161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:04.931057 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665"} err="failed to get container status \"315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665\": rpc error: code = NotFound desc = could not find container \"315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665\": container with ID starting with 315312d9978620c21d1dd3846b254d64e3c1cca5616c4249f65d12a91fd87665 not found: ID does not exist" Nov 01 15:28:05.083587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.083557 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7"] Nov 01 15:28:05.131903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.131865 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6ff6c87fcc-vtgh7"] Nov 01 15:28:05.469047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.469013 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:05.627964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.627913 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" podUID="21a7a7d1-2155-40bd-8866-66924764d547" containerName="payload-pre-processing" containerID="cri-o://ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6" gracePeriod=25 Nov 01 15:28:05.901037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.901012 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:28:05.924062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.924034 5473 generic.go:356] "Generic (PLEG): container finished" podID="21a7a7d1-2155-40bd-8866-66924764d547" containerID="ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6" exitCode=0 Nov 01 15:28:05.924516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.924079 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" Nov 01 15:28:05.924516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.924112 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" event={"ID":"21a7a7d1-2155-40bd-8866-66924764d547","Type":"ContainerDied","Data":"ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6"} Nov 01 15:28:05.924516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.924165 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n" event={"ID":"21a7a7d1-2155-40bd-8866-66924764d547","Type":"ContainerDied","Data":"85ce3d5e542056e363c7ff18d9a1d8b4e8b1a093de97fb56990e670e5976e0b8"} Nov 01 15:28:05.924516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.924189 5473 scope.go:122] "RemoveContainer" containerID="ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6" Nov 01 15:28:05.938003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.937979 5473 scope.go:122] "RemoveContainer" containerID="ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6" Nov 01 15:28:05.938449 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:05.938420 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6\": container with ID starting with ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6 not found: ID does not exist" containerID="ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6" Nov 01 15:28:05.938564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.938459 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6"} err="failed to get container status \"ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6\": rpc error: code = NotFound desc = could not find container \"ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6\": container with ID starting with ca4a16677431ffb242528d0fea3a6911790b52ec4558d95b3860c24d724607a6 not found: ID does not exist" Nov 01 15:28:05.980342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.980307 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume\") pod \"21a7a7d1-2155-40bd-8866-66924764d547\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " Nov 01 15:28:05.980506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.980425 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8\" (UniqueName: \"kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8\") pod \"21a7a7d1-2155-40bd-8866-66924764d547\" (UID: \"21a7a7d1-2155-40bd-8866-66924764d547\") " Nov 01 15:28:05.980679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.980656 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume" pod "21a7a7d1-2155-40bd-8866-66924764d547" (UID: "21a7a7d1-2155-40bd-8866-66924764d547"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:28:05.980929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.980904 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a7a7d1-2155-40bd-8866-66924764d547-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:05.982587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:05.982563 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8" pod "21a7a7d1-2155-40bd-8866-66924764d547" (UID: "21a7a7d1-2155-40bd-8866-66924764d547"). InnerVolumeSpecName "kube-api-access-qfzk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:28:06.082471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:06.082440 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzk8\" (UniqueName: \"kubernetes.io/projected/21a7a7d1-2155-40bd-8866-66924764d547-kube-api-access-qfzk8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:06.347434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:06.347348 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n"] Nov 01 15:28:06.350906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:06.350870 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89e5ef-6db56f9c9-rbv9n"] Nov 01 15:28:06.577256 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:06.577210 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:06.747583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:06.747504 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:06.966456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:06.966425 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:07.099620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.099586 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a7a7d1-2155-40bd-8866-66924764d547" path="/var/lib/kubelet/pods/21a7a7d1-2155-40bd-8866-66924764d547/volumes" Nov 01 15:28:07.100641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.100603 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c767e474-9ab0-4e34-a6ed-92656dd6f156" path="/var/lib/kubelet/pods/c767e474-9ab0-4e34-a6ed-92656dd6f156/volumes" Nov 01 15:28:07.135401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.135371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:07.348973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.348939 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq"] Nov 01 15:28:07.349618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349603 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" Nov 01 15:28:07.349618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349618 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb94eef-c4a0-4f62-aa95-3b1b7f69f803" containerName="istio-proxy" Nov 01 15:28:07.349764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349630 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c767e474-9ab0-4e34-a6ed-92656dd6f156" containerName="payload-processing" Nov 01 15:28:07.349764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349635 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c767e474-9ab0-4e34-a6ed-92656dd6f156" containerName="payload-processing" Nov 01 15:28:07.349764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349650 5473 cpu_manager.go:414] "containerMap: removing container" podUID="21a7a7d1-2155-40bd-8866-66924764d547" containerName="payload-pre-processing" Nov 01 15:28:07.349764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349656 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a7a7d1-2155-40bd-8866-66924764d547" containerName="payload-pre-processing" Nov 01 15:28:07.349891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349781 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c767e474-9ab0-4e34-a6ed-92656dd6f156" containerName="payload-processing" Nov 01 15:28:07.349891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.349790 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a7a7d1-2155-40bd-8866-66924764d547" containerName="payload-pre-processing" Nov 01 15:28:07.354358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.354339 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.408978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.408947 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq"] Nov 01 15:28:07.454571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.454536 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-96165a\"" Nov 01 15:28:07.454571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.454555 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-96165a-dockercfg-zlpn5\"" Nov 01 15:28:07.516236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.508678 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8fs9\" (UniqueName: \"kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9\") pod \"payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.516236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.508803 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.520726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.520694 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg"] Nov 01 15:28:07.533241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.527512 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.611954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.611865 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8fs9\" (UniqueName: \"kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9\") pod \"payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.616656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.616614 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.616802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.611908 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg"] Nov 01 15:28:07.618075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.618046 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.666014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.665982 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:07.668680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.668654 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8fs9\" (UniqueName: \"kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9\") pod \"payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:07.718454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.718414 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnlt7\" (UniqueName: \"kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7\") pod \"payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.718641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.718610 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.820125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.820082 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.820324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.820303 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnlt7\" (UniqueName: \"kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7\") pod \"payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.820862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.820844 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.877040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.876971 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnlt7\" (UniqueName: \"kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7\") pod \"payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.889979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.889953 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:07.960253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.959724 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:07.968633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:07.968600 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:08.274864 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:28:08.274462 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe101fc_9a38_4bcf_bf63_002bbf93709d.slice/crio-a62fc2024ec0b708a20c081b0ddd3420e29d412aa81c48a3148e6e6ce481a1ba WatchSource:0}: Error finding container a62fc2024ec0b708a20c081b0ddd3420e29d412aa81c48a3148e6e6ce481a1ba: Status 404 returned error can't find the container with id a62fc2024ec0b708a20c081b0ddd3420e29d412aa81c48a3148e6e6ce481a1ba Nov 01 15:28:08.277084 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.277062 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg"] Nov 01 15:28:08.383351 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:28:08.383318 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94500465_cb49_41e5_9378_71b86396fda9.slice/crio-2b9164b48f6126086d204251c672895ee3ceec9f0850a47677cd790cee2065ad WatchSource:0}: Error finding container 2b9164b48f6126086d204251c672895ee3ceec9f0850a47677cd790cee2065ad: Status 404 returned error can't find the container with id 2b9164b48f6126086d204251c672895ee3ceec9f0850a47677cd790cee2065ad Nov 01 15:28:08.385670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.385650 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq"] Nov 01 15:28:08.417473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.417445 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:08.958330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.958206 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" event={"ID":"abe101fc-9a38-4bcf-bf63-002bbf93709d","Type":"ContainerStarted","Data":"ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f"} Nov 01 15:28:08.958330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.958293 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" event={"ID":"abe101fc-9a38-4bcf-bf63-002bbf93709d","Type":"ContainerStarted","Data":"a62fc2024ec0b708a20c081b0ddd3420e29d412aa81c48a3148e6e6ce481a1ba"} Nov 01 15:28:08.958592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.958412 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:08.959868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.959840 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" event={"ID":"94500465-cb49-41e5-9378-71b86396fda9","Type":"ContainerStarted","Data":"de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93"} Nov 01 15:28:08.959978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.959886 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" event={"ID":"94500465-cb49-41e5-9378-71b86396fda9","Type":"ContainerStarted","Data":"2b9164b48f6126086d204251c672895ee3ceec9f0850a47677cd790cee2065ad"} Nov 01 15:28:08.959978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:08.959928 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:09.075543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:09.075493 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" podStartSLOduration=2.075478857 podStartE2EDuration="2.075478857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:28:09.071259776 +0000 UTC m=+4228.129411625" watchObservedRunningTime="2026-11-01 15:28:09.075478857 +0000 UTC m=+4228.133630707" Nov 01 15:28:09.110732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:09.110702 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:09.187667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:09.187611 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" podStartSLOduration=2.187594105 podStartE2EDuration="2.187594105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:28:09.183397595 +0000 UTC m=+4228.241549445" watchObservedRunningTime="2026-11-01 15:28:09.187594105 +0000 UTC m=+4228.245745955" Nov 01 15:28:11.319746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:11.319716 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:12.319474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:12.319440 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:14.855492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:14.855456 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:15.678774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:15.678740 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh"] Nov 01 15:28:16.045200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:16.045161 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:16.210811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:16.210772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:16.941345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:16.941311 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:17.636465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.636429 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz"] Nov 01 15:28:17.640800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.640782 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.693333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.693306 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 15:28:17.693333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.693303 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 15:28:17.693333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.693306 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 15:28:17.734636 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.734608 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lmbst\"" Nov 01 15:28:17.747593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.747566 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz"] Nov 01 15:28:17.765489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765461 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.765656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765542 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.765656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765584 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.765818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765670 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.765818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765735 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwk7\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.765818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765806 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.765999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.765833 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.866756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.866724 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.866930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.866769 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.866930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.866887 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.867063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.866980 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.867063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.867008 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.867179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.867085 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.867179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.867125 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwk7\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.867794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.867763 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.869396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.869375 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.869522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.869414 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.869598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.869575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.869666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.869642 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.925154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.925087 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.925325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.925302 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwk7\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-f8fzz\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:17.954632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:17.954606 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:18.372350 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:28:18.372319 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e831cf_4fa8_4ec0_8365_8b9c250a9af9.slice/crio-91a69b127d66eedef9cc5987a5bf80d21ac726cdf6e8c2947949177c37477644 WatchSource:0}: Error finding container 91a69b127d66eedef9cc5987a5bf80d21ac726cdf6e8c2947949177c37477644: Status 404 returned error can't find the container with id 91a69b127d66eedef9cc5987a5bf80d21ac726cdf6e8c2947949177c37477644 Nov 01 15:28:18.375519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:18.375481 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:28:18.375618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:18.375563 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:28:18.378430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:18.378407 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz"] Nov 01 15:28:19.088447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.088411 5473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-f8fzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:28:19.088924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.088469 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:28:19.091374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.091351 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:19.091508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.091395 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" event={"ID":"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9","Type":"ContainerStarted","Data":"3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927"} Nov 01 15:28:19.091508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.091412 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" event={"ID":"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9","Type":"ContainerStarted","Data":"91a69b127d66eedef9cc5987a5bf80d21ac726cdf6e8c2947949177c37477644"} Nov 01 15:28:19.204671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.204607 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" podStartSLOduration=2.204590976 podStartE2EDuration="2.204590976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:28:19.203495293 +0000 UTC m=+4238.261647145" watchObservedRunningTime="2026-11-01 15:28:19.204590976 +0000 UTC m=+4238.262742827" Nov 01 15:28:19.971016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.970985 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:28:19.971200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:19.971059 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:28:20.097135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:20.097098 5473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-f8fzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:28:20.097699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:20.097165 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:28:20.627304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:20.627196 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:20.679132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:20.679088 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" podUID="380357dc-ed41-4a76-b02c-c3a0641ac252" containerName="payload-processing" containerID="cri-o://c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51" gracePeriod=25 Nov 01 15:28:20.951889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:20.951864 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:28:21.015932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.015892 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume\") pod \"380357dc-ed41-4a76-b02c-c3a0641ac252\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " Nov 01 15:28:21.016130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.016038 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x\" (UniqueName: \"kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x\") pod \"380357dc-ed41-4a76-b02c-c3a0641ac252\" (UID: \"380357dc-ed41-4a76-b02c-c3a0641ac252\") " Nov 01 15:28:21.016307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.016283 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume" pod "380357dc-ed41-4a76-b02c-c3a0641ac252" (UID: "380357dc-ed41-4a76-b02c-c3a0641ac252"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:28:21.016649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.016628 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/380357dc-ed41-4a76-b02c-c3a0641ac252-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:21.018112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.018088 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x" pod "380357dc-ed41-4a76-b02c-c3a0641ac252" (UID: "380357dc-ed41-4a76-b02c-c3a0641ac252"). InnerVolumeSpecName "kube-api-access-jls5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:28:21.106120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.106088 5473 generic.go:356] "Generic (PLEG): container finished" podID="380357dc-ed41-4a76-b02c-c3a0641ac252" containerID="c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51" exitCode=0 Nov 01 15:28:21.106584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.106148 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" Nov 01 15:28:21.106584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.106168 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" event={"ID":"380357dc-ed41-4a76-b02c-c3a0641ac252","Type":"ContainerDied","Data":"c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51"} Nov 01 15:28:21.106584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.106237 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh" event={"ID":"380357dc-ed41-4a76-b02c-c3a0641ac252","Type":"ContainerDied","Data":"2df664f665c935f54aa53db54123ce5be0d6df1c3212c10f8499524f0bc61b94"} Nov 01 15:28:21.106584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.106279 5473 scope.go:122] "RemoveContainer" containerID="c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51" Nov 01 15:28:21.118020 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.117995 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jls5x\" (UniqueName: \"kubernetes.io/projected/380357dc-ed41-4a76-b02c-c3a0641ac252-kube-api-access-jls5x\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:21.118944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.118927 5473 scope.go:122] "RemoveContainer" containerID="c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51" Nov 01 15:28:21.119253 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:21.119227 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51\": container with ID starting with c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51 not found: ID does not exist" containerID="c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51" Nov 01 15:28:21.119351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.119257 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51"} err="failed to get container status \"c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51\": rpc error: code = NotFound desc = could not find container \"c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51\": container with ID starting with c2873a771e94bc1d28aeb91d630b4939d7806cd2717511723ccb197cf7fd4d51 not found: ID does not exist" Nov 01 15:28:21.273200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.273163 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh"] Nov 01 15:28:21.276741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.276713 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-04aa47-d46fb99f7-nd6vh"] Nov 01 15:28:21.796599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:21.796563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:23.083509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:23.083475 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380357dc-ed41-4a76-b02c-c3a0641ac252" path="/var/lib/kubelet/pods/380357dc-ed41-4a76-b02c-c3a0641ac252/volumes" Nov 01 15:28:23.097753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:23.097728 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:28:26.711097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:26.711057 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:31.095659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:31.095622 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:31.110602 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:31.110570 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:28:34.934976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:34.934942 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:36.003276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:36.003173 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:36.173651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:36.173617 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:36.486054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:36.486022 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:36.638916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:36.638885 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:36.687774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:36.687742 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb"] Nov 01 15:28:36.738294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:36.738180 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-775095-gtwlb"] Nov 01 15:28:37.016029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:37.015992 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:37.118542 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:37.116589 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0f8ed6-a18c-4cc5-b863-990c95f3ef51" path="/var/lib/kubelet/pods/af0f8ed6-a18c-4cc5-b863-990c95f3ef51/volumes" Nov 01 15:28:39.355249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:39.354514 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:39.532390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:39.532327 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:41.575820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:41.575778 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:44.386534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:44.386501 5473 scope.go:122] "RemoveContainer" containerID="8ed1cc9ed1e7a99656aaca19b7d7193ee7e2912f6a9ea0da03381d1f831119d8" Nov 01 15:28:45.988855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:45.988822 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:49.783045 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:49.783001 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:51.358706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.358674 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:51.532129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.532091 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:51.543838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.543794 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l"] Nov 01 15:28:51.591989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.591953 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5feb76-kgm2l"] Nov 01 15:28:51.752313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.752226 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9"] Nov 01 15:28:51.752548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.752520 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" containerID="cri-o://6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7" gracePeriod=30 Nov 01 15:28:51.817149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:51.817122 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:53.082536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:53.082500 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da23cdbf-7d67-4d92-9608-e4a2e04cfe4a" path="/var/lib/kubelet/pods/da23cdbf-7d67-4d92-9608-e4a2e04cfe4a/volumes" Nov 01 15:28:54.094626 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:54.094585 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:54.277101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:54.277071 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:56.360854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:56.360806 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:57.020198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.020138 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:28:57.130264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130226 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130287 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130336 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130364 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130490 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130570 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130637 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:28:57.130726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130649 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:28:57.130854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130727 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130771 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.130854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130765 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:28:57.130854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130811 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert\") pod \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\" (UID: \"2940993e-3d29-49ca-b4bf-01f2fd2a88f5\") " Nov 01 15:28:57.131070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.130928 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:28:57.131669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.131405 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:28:57.131669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.131516 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.131669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.131540 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.131669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.131554 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.131669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.131568 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.131669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.131581 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.133883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.133857 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:28:57.133883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.133871 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:28:57.134067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.133989 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "kube-api-access-2gw6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:28:57.134067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.134012 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token" pod "2940993e-3d29-49ca-b4bf-01f2fd2a88f5" (UID: "2940993e-3d29-49ca-b4bf-01f2fd2a88f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:28:57.232904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.232868 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.232904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.232899 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-kube-api-access-2gw6b\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.232904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.232911 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.233253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.232919 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2940993e-3d29-49ca-b4bf-01f2fd2a88f5-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:28:57.456049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.455951 5473 generic.go:356] "Generic (PLEG): container finished" podID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerID="6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7" exitCode=0 Nov 01 15:28:57.456049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.456034 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" Nov 01 15:28:57.456584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.456040 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" event={"ID":"2940993e-3d29-49ca-b4bf-01f2fd2a88f5","Type":"ContainerDied","Data":"6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7"} Nov 01 15:28:57.456584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.456104 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9" event={"ID":"2940993e-3d29-49ca-b4bf-01f2fd2a88f5","Type":"ContainerDied","Data":"9d1ffea550d86ff770339ccf1e99b712cbeebea348f7a1aa41cdc92ebee5c050"} Nov 01 15:28:57.456584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.456134 5473 scope.go:122] "RemoveContainer" containerID="6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7" Nov 01 15:28:57.467938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.467917 5473 scope.go:122] "RemoveContainer" containerID="f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63" Nov 01 15:28:57.477950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.477933 5473 scope.go:122] "RemoveContainer" containerID="6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7" Nov 01 15:28:57.478291 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:57.478271 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7\": container with ID starting with 6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7 not found: ID does not exist" containerID="6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7" Nov 01 15:28:57.478338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.478300 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7"} err="failed to get container status \"6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7\": rpc error: code = NotFound desc = could not find container \"6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7\": container with ID starting with 6357270a85e2daa6d3155d69c7dbbbe54f42ae50d488182fecf51e0fdc22aec7 not found: ID does not exist" Nov 01 15:28:57.478338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.478320 5473 scope.go:122] "RemoveContainer" containerID="f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63" Nov 01 15:28:57.478583 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:28:57.478565 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63\": container with ID starting with f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63 not found: ID does not exist" containerID="f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63" Nov 01 15:28:57.478627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.478590 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63"} err="failed to get container status \"f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63\": rpc error: code = NotFound desc = could not find container \"f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63\": container with ID starting with f34eacafc88f968e442bd6f24c7c74ec73b29876fe84a8c8f82dbbe3d0878e63 not found: ID does not exist" Nov 01 15:28:57.622547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.622509 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9"] Nov 01 15:28:57.672947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:57.672915 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5feb76-openshift-default-cd7f54497-vhwv9"] Nov 01 15:28:58.647385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:58.647346 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:58.813898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:58.813852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:28:59.097797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:28:59.097759 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" path="/var/lib/kubelet/pods/2940993e-3d29-49ca-b4bf-01f2fd2a88f5/volumes" Nov 01 15:29:00.205245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:00.205185 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:00.385945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:00.385913 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:01.109888 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:29:01.109842 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:29:01.389609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:01.389517 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:05.782016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:05.781981 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:05.986385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:05.986352 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:10.335722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:10.335684 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:10.492969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:10.492938 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:14.727717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:14.727670 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:16.124590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:16.124561 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:18.534898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:18.534861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:20.137521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:20.137485 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:22.136861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:22.136818 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:22.585770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:22.585734 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:23.536133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:23.536100 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:23.705381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:23.705341 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:24.735442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:24.735407 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:24.908629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:24.908589 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:25.101738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:25.101707 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:26.758966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.758931 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk"] Nov 01 15:29:26.759834 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.759810 5473 cpu_manager.go:414] "containerMap: removing container" podUID="380357dc-ed41-4a76-b02c-c3a0641ac252" containerName="payload-processing" Nov 01 15:29:26.759952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.759835 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380357dc-ed41-4a76-b02c-c3a0641ac252" containerName="payload-processing" Nov 01 15:29:26.759952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.759862 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" Nov 01 15:29:26.759952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.759870 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" Nov 01 15:29:26.760134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.760057 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="380357dc-ed41-4a76-b02c-c3a0641ac252" containerName="payload-processing" Nov 01 15:29:26.760134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.760078 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" Nov 01 15:29:26.763838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.763820 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.819948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.819920 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk"] Nov 01 15:29:26.841671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.841634 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:29:26.855637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.855603 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.855892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.855871 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5z4m\" (UniqueName: \"kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.855971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.855939 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.865409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.865386 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:29:26.957030 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.956999 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5z4m\" (UniqueName: \"kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.957199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.957051 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.957199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.957119 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.957785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.957761 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:26.959599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:26.959575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:27.014056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.014030 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5z4m\" (UniqueName: \"kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m\") pod \"maas-api-revoke-sub-febbfd88-98rqk\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:27.078856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.078822 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:27.183593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.183477 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:27.352954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.352849 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:27.458872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.458831 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk"] Nov 01 15:29:27.794998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.794961 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" event={"ID":"302e2c9b-8141-4828-a92c-e55de67e40c0","Type":"ContainerStarted","Data":"b64f041dce5590fd475d1183298b8c5b46169861c4417c85225ae59fd142be6b"} Nov 01 15:29:27.795386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.795022 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" event={"ID":"302e2c9b-8141-4828-a92c-e55de67e40c0","Type":"ContainerStarted","Data":"fbaaa49a64ed172e5f964e70816222b2d4d716d56d9ca09d655d92a5621a2066"} Nov 01 15:29:27.910120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:27.910027 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" podStartSLOduration=1.910012888 podStartE2EDuration="1.910012888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:29:27.908501564 +0000 UTC m=+4306.966653396" watchObservedRunningTime="2026-11-01 15:29:27.910012888 +0000 UTC m=+4306.968164737" Nov 01 15:29:28.804830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:28.804796 5473 generic.go:356] "Generic (PLEG): container finished" podID="302e2c9b-8141-4828-a92c-e55de67e40c0" containerID="b64f041dce5590fd475d1183298b8c5b46169861c4417c85225ae59fd142be6b" exitCode=0 Nov 01 15:29:28.805339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:28.804885 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" event={"ID":"302e2c9b-8141-4828-a92c-e55de67e40c0","Type":"ContainerDied","Data":"b64f041dce5590fd475d1183298b8c5b46169861c4417c85225ae59fd142be6b"} Nov 01 15:29:29.389425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:29.389388 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:29.984390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:29.984367 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:30.110474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.110384 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca\") pod \"302e2c9b-8141-4828-a92c-e55de67e40c0\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " Nov 01 15:29:30.110658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.110517 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m\" (UniqueName: \"kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m\") pod \"302e2c9b-8141-4828-a92c-e55de67e40c0\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " Nov 01 15:29:30.110765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.110749 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca\") pod \"302e2c9b-8141-4828-a92c-e55de67e40c0\" (UID: \"302e2c9b-8141-4828-a92c-e55de67e40c0\") " Nov 01 15:29:30.111286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.111253 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca" pod "302e2c9b-8141-4828-a92c-e55de67e40c0" (UID: "302e2c9b-8141-4828-a92c-e55de67e40c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:29:30.113503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.113473 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca" pod "302e2c9b-8141-4828-a92c-e55de67e40c0" (UID: "302e2c9b-8141-4828-a92c-e55de67e40c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:29:30.132442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.132404 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m" pod "302e2c9b-8141-4828-a92c-e55de67e40c0" (UID: "302e2c9b-8141-4828-a92c-e55de67e40c0"). InnerVolumeSpecName "kube-api-access-q5z4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:29:30.212953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.212919 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5z4m\" (UniqueName: \"kubernetes.io/projected/302e2c9b-8141-4828-a92c-e55de67e40c0-kube-api-access-q5z4m\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:30.212953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.212949 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:30.212953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.212961 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302e2c9b-8141-4828-a92c-e55de67e40c0-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:30.822943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.822910 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" event={"ID":"302e2c9b-8141-4828-a92c-e55de67e40c0","Type":"ContainerDied","Data":"fbaaa49a64ed172e5f964e70816222b2d4d716d56d9ca09d655d92a5621a2066"} Nov 01 15:29:30.822943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.822933 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk" Nov 01 15:29:30.822943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:30.822960 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbaaa49a64ed172e5f964e70816222b2d4d716d56d9ca09d655d92a5621a2066" Nov 01 15:29:31.109024 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:29:31.108936 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:29:32.687150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687107 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp"] Nov 01 15:29:32.687825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687806 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" Nov 01 15:29:32.687825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687824 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" Nov 01 15:29:32.687982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687835 5473 cpu_manager.go:414] "containerMap: removing container" podUID="302e2c9b-8141-4828-a92c-e55de67e40c0" containerName="revoke-keys" Nov 01 15:29:32.687982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687840 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302e2c9b-8141-4828-a92c-e55de67e40c0" containerName="revoke-keys" Nov 01 15:29:32.687982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687968 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2940993e-3d29-49ca-b4bf-01f2fd2a88f5" containerName="istio-proxy" Nov 01 15:29:32.687982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.687976 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="302e2c9b-8141-4828-a92c-e55de67e40c0" containerName="revoke-keys" Nov 01 15:29:32.691770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.691751 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.748401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.748371 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp"] Nov 01 15:29:32.793710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.793680 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:29:32.793870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.793780 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:29:32.845243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.845177 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.845402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.845359 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.845452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.845413 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2xw\" (UniqueName: \"kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.947620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.947544 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.947795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.947640 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.947795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.947683 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2xw\" (UniqueName: \"kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.948293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.948269 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:32.949937 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:32.949917 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:33.004575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.004548 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2xw\" (UniqueName: \"kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw\") pod \"maas-api-revoke-sub-fbda1dd4-f74sp\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:33.006349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.006332 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:33.373463 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:29:33.373422 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12304c5_900d_41f8_8aa2_9cc39f9da462.slice/crio-293524dfabf52988568e418da3af0f7e1274b1b8d73d74954a8975d58297f2ca WatchSource:0}: Error finding container 293524dfabf52988568e418da3af0f7e1274b1b8d73d74954a8975d58297f2ca: Status 404 returned error can't find the container with id 293524dfabf52988568e418da3af0f7e1274b1b8d73d74954a8975d58297f2ca Nov 01 15:29:33.379606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.379576 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp"] Nov 01 15:29:33.790634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.790600 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:33.849587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.849542 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" event={"ID":"c12304c5-900d-41f8-8aa2-9cc39f9da462","Type":"ContainerStarted","Data":"e4d2dae2055bf8aa4aa3ee75357caa7ebe332099ff40ded93564e00ef6065a48"} Nov 01 15:29:33.849772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.849619 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" event={"ID":"c12304c5-900d-41f8-8aa2-9cc39f9da462","Type":"ContainerStarted","Data":"293524dfabf52988568e418da3af0f7e1274b1b8d73d74954a8975d58297f2ca"} Nov 01 15:29:33.966910 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:33.966817 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" podStartSLOduration=1.966804351 podStartE2EDuration="1.966804351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:29:33.962293617 +0000 UTC m=+4313.020445466" watchObservedRunningTime="2026-11-01 15:29:33.966804351 +0000 UTC m=+4313.024956200" Nov 01 15:29:34.858452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:34.858409 5473 generic.go:356] "Generic (PLEG): container finished" podID="c12304c5-900d-41f8-8aa2-9cc39f9da462" containerID="e4d2dae2055bf8aa4aa3ee75357caa7ebe332099ff40ded93564e00ef6065a48" exitCode=0 Nov 01 15:29:34.858855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:34.858542 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" event={"ID":"c12304c5-900d-41f8-8aa2-9cc39f9da462","Type":"ContainerDied","Data":"e4d2dae2055bf8aa4aa3ee75357caa7ebe332099ff40ded93564e00ef6065a48"} Nov 01 15:29:36.032386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.032361 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:36.188075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.187980 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw\" (UniqueName: \"kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw\") pod \"c12304c5-900d-41f8-8aa2-9cc39f9da462\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " Nov 01 15:29:36.188284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.188164 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca\") pod \"c12304c5-900d-41f8-8aa2-9cc39f9da462\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " Nov 01 15:29:36.188378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.188299 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca\") pod \"c12304c5-900d-41f8-8aa2-9cc39f9da462\" (UID: \"c12304c5-900d-41f8-8aa2-9cc39f9da462\") " Nov 01 15:29:36.188719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.188690 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca" pod "c12304c5-900d-41f8-8aa2-9cc39f9da462" (UID: "c12304c5-900d-41f8-8aa2-9cc39f9da462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:29:36.189065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.189034 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:36.190311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.190287 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca" pod "c12304c5-900d-41f8-8aa2-9cc39f9da462" (UID: "c12304c5-900d-41f8-8aa2-9cc39f9da462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:29:36.190571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.190543 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw" pod "c12304c5-900d-41f8-8aa2-9cc39f9da462" (UID: "c12304c5-900d-41f8-8aa2-9cc39f9da462"). InnerVolumeSpecName "kube-api-access-cz2xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:29:36.290780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.290746 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12304c5-900d-41f8-8aa2-9cc39f9da462-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:36.290780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.290775 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2xw\" (UniqueName: \"kubernetes.io/projected/c12304c5-900d-41f8-8aa2-9cc39f9da462-kube-api-access-cz2xw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:36.876689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.876659 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" Nov 01 15:29:36.876866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.876696 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp" event={"ID":"c12304c5-900d-41f8-8aa2-9cc39f9da462","Type":"ContainerDied","Data":"293524dfabf52988568e418da3af0f7e1274b1b8d73d74954a8975d58297f2ca"} Nov 01 15:29:36.876866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:36.876744 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="293524dfabf52988568e418da3af0f7e1274b1b8d73d74954a8975d58297f2ca" Nov 01 15:29:37.420302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:37.420255 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:37.612726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:37.612682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:39.229232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:39.229183 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:40.516731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:40.516696 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:40.938361 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:40.938280 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758"] Nov 01 15:29:40.938543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:40.938519 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" containerID="cri-o://350793c3349b0f0eb385adfa9dc384b12d85c5076dcad22f54fa6bfea4985032" gracePeriod=30 Nov 01 15:29:40.969609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:40.969574 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:41.890808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:41.890775 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:42.062025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:42.061993 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:42.387868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:42.387838 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:43.300969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:43.300937 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:43.683288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:43.683186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:44.133507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:44.133477 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:45.043057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:45.043022 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:45.249764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:45.249713 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:47.053300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.039309 5473 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded" Nov 01 15:29:47.124241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.116233 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:29:47.242647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.240518 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz" podUID="8dcdd7f9-e49a-4a88-a4af-d888fc97551d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:29:47.278731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.278363 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:29:47.293683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.293303 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:29:47.546238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.544642 5473 generic.go:356] "Generic (PLEG): container finished" podID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerID="350793c3349b0f0eb385adfa9dc384b12d85c5076dcad22f54fa6bfea4985032" exitCode=0 Nov 01 15:29:47.546417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.546267 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" event={"ID":"d0c919c2-feaf-4ae3-ba57-48c1d3f41261","Type":"ContainerDied","Data":"350793c3349b0f0eb385adfa9dc384b12d85c5076dcad22f54fa6bfea4985032"} Nov 01 15:29:47.546417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.546338 5473 scope.go:122] "RemoveContainer" containerID="fea23a62621cffcb19105fef56ace4319276b434eb71ad1a6ebcdaf0d07c5e87" Nov 01 15:29:47.560184 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.560161 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:29:47.598327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.598290 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" probeResult="failure" output="Get \"https://10.134.0.112:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.733737 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.733816 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.733869 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.733908 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.733940 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.733976 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.734041 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.734296 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.749161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.734340 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs\") pod \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\" (UID: \"d0c919c2-feaf-4ae3-ba57-48c1d3f41261\") " Nov 01 15:29:47.767246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.764330 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:29:47.769355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.769305 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:29:47.774367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.774312 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:29:47.778355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.778311 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:29:47.778928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.778907 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:29:47.834053 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.833974 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:29:47.844323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.837650 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.844323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.837683 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.844323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.837699 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.844323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.837713 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.844323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.837726 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.844323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.837737 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.882798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.882743 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:29:47.883336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.883262 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "kube-api-access-b5jqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:29:47.884165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.884121 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo" pod "d0c919c2-feaf-4ae3-ba57-48c1d3f41261" (UID: "d0c919c2-feaf-4ae3-ba57-48c1d3f41261"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:29:47.945705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.945623 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.945705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.945659 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:47.945705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:47.945676 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5jqw\" (UniqueName: \"kubernetes.io/projected/d0c919c2-feaf-4ae3-ba57-48c1d3f41261-kube-api-access-b5jqw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:29:48.493678 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.493576 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:48.558600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.558564 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" Nov 01 15:29:48.558761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.558593 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758" event={"ID":"d0c919c2-feaf-4ae3-ba57-48c1d3f41261","Type":"ContainerDied","Data":"f2ec0a96686d14ee07fea7e9ae634569726c69c0dfe9fbb27a396181e6438ae8"} Nov 01 15:29:48.558761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.558652 5473 scope.go:122] "RemoveContainer" containerID="350793c3349b0f0eb385adfa9dc384b12d85c5076dcad22f54fa6bfea4985032" Nov 01 15:29:48.705445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.705401 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:48.864671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.864630 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758"] Nov 01 15:29:48.869732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.869704 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-04aa47-openshift-default-5658ccb679-dv758"] Nov 01 15:29:48.884892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:48.884867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:49.058048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:49.058014 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:49.083079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:49.083053 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" path="/var/lib/kubelet/pods/d0c919c2-feaf-4ae3-ba57-48c1d3f41261/volumes" Nov 01 15:29:49.666486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:49.666437 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:53.127198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:53.127168 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:54.609932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.609858 5473 generic.go:356] "Generic (PLEG): container finished" podID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerID="2358c93315e86b82a3574abc11ed577a7be55c205df05bc2cb7e36b5580545a1" exitCode=0 Nov 01 15:29:54.610433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.609934 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" event={"ID":"88f9ff98-2bbc-4126-b236-9bcc598d02d6","Type":"ContainerDied","Data":"2358c93315e86b82a3574abc11ed577a7be55c205df05bc2cb7e36b5580545a1"} Nov 01 15:29:54.610433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.610408 5473 scope.go:122] "RemoveContainer" containerID="2358c93315e86b82a3574abc11ed577a7be55c205df05bc2cb7e36b5580545a1" Nov 01 15:29:54.611971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.611941 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:29:54.612076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.612017 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:29:54.612076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.612050 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:29:54.753383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.753341 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:54.926020 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:54.925932 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:55.619037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:55.619001 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" event={"ID":"88f9ff98-2bbc-4126-b236-9bcc598d02d6","Type":"ContainerStarted","Data":"55591be9f016706e984e911b34cc6faa497a1c1ca775b220c9080d4acaeb205a"} Nov 01 15:29:56.076960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:56.076925 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:29:56.077183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:56.077142 5473 patch_prober.go:29] interesting pod/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 15:29:56.077313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:56.077199 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 15:29:57.076977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:57.076935 5473 patch_prober.go:29] interesting pod/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 15:29:57.077412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:57.077003 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 15:29:57.479113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:57.479032 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:29:58.076992 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:58.076959 5473 patch_prober.go:29] interesting pod/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 15:29:58.077416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:58.077018 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 15:29:59.076455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:59.076420 5473 patch_prober.go:29] interesting pod/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 15:29:59.076666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:29:59.076479 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 15:30:00.076256 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.076191 5473 patch_prober.go:29] interesting pod/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body= Nov 01 15:30:00.076662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.076282 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" Nov 01 15:30:00.173038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173003 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86"] Nov 01 15:30:00.173704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173688 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c12304c5-900d-41f8-8aa2-9cc39f9da462" containerName="revoke-keys" Nov 01 15:30:00.173778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173704 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12304c5-900d-41f8-8aa2-9cc39f9da462" containerName="revoke-keys" Nov 01 15:30:00.173778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173722 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" Nov 01 15:30:00.173778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173727 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" Nov 01 15:30:00.173884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173822 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" Nov 01 15:30:00.173884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173836 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" Nov 01 15:30:00.173884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.173850 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12304c5-900d-41f8-8aa2-9cc39f9da462" containerName="revoke-keys" Nov 01 15:30:00.176480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.176461 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g"] Nov 01 15:30:00.176600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.176580 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.177148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.177125 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" Nov 01 15:30:00.177148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.177147 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c919c2-feaf-4ae3-ba57-48c1d3f41261" containerName="istio-proxy" Nov 01 15:30:00.179463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.179448 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.185094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.185079 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl"] Nov 01 15:30:00.187896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.187873 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.188057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.188037 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9"] Nov 01 15:30:00.190738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.190719 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.222070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222048 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.222166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222094 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2ssr\" (UniqueName: \"kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.222264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222250 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.222320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222278 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtl97\" (UniqueName: \"kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97\") pod \"maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.222369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222331 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.222423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222410 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmm7\" (UniqueName: \"kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.222887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.222874 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt"] Nov 01 15:30:00.234639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.234595 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:30:00.250919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.250890 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5"] Nov 01 15:30:00.251570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.251332 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.255406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.255389 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8"] Nov 01 15:30:00.255532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.255510 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.259044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.259021 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.272057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.272028 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9"] Nov 01 15:30:00.278085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.278062 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.279613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.279574 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk"] Nov 01 15:30:00.284286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.283768 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.287283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.287207 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd"] Nov 01 15:30:00.290265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.290246 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.292425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.292400 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln"] Nov 01 15:30:00.296365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.296346 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.296478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.296412 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86"] Nov 01 15:30:00.300544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.300526 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g"] Nov 01 15:30:00.305649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.305630 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt"] Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324470 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324519 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtl97\" (UniqueName: \"kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97\") pod \"maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324565 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324614 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt\") pod \"maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324666 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324697 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmm7\" (UniqueName: \"kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324753 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324790 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qhk\" (UniqueName: \"kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324842 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlgl2\" (UniqueName: \"kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324945 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85dc8\" (UniqueName: \"kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.324988 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.325017 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.325065 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2ssr\" (UniqueName: \"kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.326090 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.326358 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.326391 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwdv\" (UniqueName: \"kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.326423 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nj4k\" (UniqueName: \"kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.326451 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.328583 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.331533 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.341429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.338392 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl"] Nov 01 15:30:00.379558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.352652 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9"] Nov 01 15:30:00.379558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.370987 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5"] Nov 01 15:30:00.379558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.376057 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk"] Nov 01 15:30:00.381710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.381636 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd"] Nov 01 15:30:00.388253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.388194 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2ssr\" (UniqueName: \"kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.392754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.392734 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtl97\" (UniqueName: \"kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97\") pod \"maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.392901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.392877 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmm7\" (UniqueName: \"kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7\") pod \"maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.421795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.421767 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9"] Nov 01 15:30:00.440265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.436044 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85dc8\" (UniqueName: \"kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.446528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.446498 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.446788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.446766 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.446924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.446908 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwdv\" (UniqueName: \"kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.447037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.447020 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.447169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.447153 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nj4k\" (UniqueName: \"kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.447304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.447288 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.450070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.448497 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.450070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.449127 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.450070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.450025 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.454578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.447477 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2v7t\" (UniqueName: \"kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.454747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.454727 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k9t7\" (UniqueName: \"kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7\") pod \"maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.454915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.454898 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.455048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.455033 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt\") pod \"maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.455196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.455181 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.455343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.455326 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.455466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.455452 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.455589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.455575 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qhk\" (UniqueName: \"kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.455736 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.455720 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlgl2\" (UniqueName: \"kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.457095 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.457072 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.459698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.458902 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.459698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.459645 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.460892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.460873 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8"] Nov 01 15:30:00.465360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.465340 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln"] Nov 01 15:30:00.490556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.490533 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:00.497317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.497298 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85dc8\" (UniqueName: \"kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.499139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.499121 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:00.504674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.504651 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwdv\" (UniqueName: \"kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.507512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.507490 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:00.507952 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.507926 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nj4k\" (UniqueName: \"kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k\") pod \"maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.512788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.512766 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlgl2\" (UniqueName: \"kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.513579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.513464 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt\") pod \"maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.515782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.515762 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qhk\" (UniqueName: \"kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.515922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.515908 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:00.561247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.560292 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.561247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.560397 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2v7t\" (UniqueName: \"kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.561247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.560431 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k9t7\" (UniqueName: \"kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7\") pod \"maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.561247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.560496 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.562948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.561763 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.562948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.562470 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.581797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.578993 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:00.600771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.600741 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:00.612383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.612042 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:00.622957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.622780 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k9t7\" (UniqueName: \"kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7\") pod \"maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.623416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.623236 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:00.628713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.624547 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2v7t\" (UniqueName: \"kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t\") pod \"maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.639933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.639900 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:00.660564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.660535 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:00.673354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.671003 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:00.911424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:00.909555 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86"] Nov 01 15:30:00.915688 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:30:00.915662 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e528eda_7ad6_4f05_b88d_a19125f1eacc.slice/crio-c3e15e877db0cdc029b7bd43fd31c64e38a9d1b36c29080ba9aad1994866fa1b WatchSource:0}: Error finding container c3e15e877db0cdc029b7bd43fd31c64e38a9d1b36c29080ba9aad1994866fa1b: Status 404 returned error can't find the container with id c3e15e877db0cdc029b7bd43fd31c64e38a9d1b36c29080ba9aad1994866fa1b Nov 01 15:30:01.086424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.086390 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:30:01.088665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.088638 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:30:01.089390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.089372 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:30:01.112461 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:30:01.112418 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e434e8_3a12_4943_a4fd_579026276228.slice/crio-f1d8c79cca17ad5a2d42b951b1cba8ab1a7c15be2df4b84389416298874ef34e WatchSource:0}: Error finding container f1d8c79cca17ad5a2d42b951b1cba8ab1a7c15be2df4b84389416298874ef34e: Status 404 returned error can't find the container with id f1d8c79cca17ad5a2d42b951b1cba8ab1a7c15be2df4b84389416298874ef34e Nov 01 15:30:01.116593 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:01.116566 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:30:01.117830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.117807 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g"] Nov 01 15:30:01.171784 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:30:01.171755 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48423bb2_dc91_41b2_9ea5_ac1770ea3312.slice/crio-4b4a2fdbb2be1783cc984399c21e343ff4fa32e9b7e40e710b0794cd626b89ce WatchSource:0}: Error finding container 4b4a2fdbb2be1783cc984399c21e343ff4fa32e9b7e40e710b0794cd626b89ce: Status 404 returned error can't find the container with id 4b4a2fdbb2be1783cc984399c21e343ff4fa32e9b7e40e710b0794cd626b89ce Nov 01 15:30:01.172437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.172411 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk"] Nov 01 15:30:01.177873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.177857 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd"] Nov 01 15:30:01.191017 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:30:01.190991 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d54414d_4f51_4dec_a597_da362f0958ca.slice/crio-820cc6531ea5ea4528f5d0dfbea90a6e20473b13b98d371c1be1591c47e88d67 WatchSource:0}: Error finding container 820cc6531ea5ea4528f5d0dfbea90a6e20473b13b98d371c1be1591c47e88d67: Status 404 returned error can't find the container with id 820cc6531ea5ea4528f5d0dfbea90a6e20473b13b98d371c1be1591c47e88d67 Nov 01 15:30:01.191538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.191517 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9"] Nov 01 15:30:01.208594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.208571 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl"] Nov 01 15:30:01.210563 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:30:01.210536 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaddc32d_14c2_4d45_947f_ff47b9f154d3.slice/crio-72b81dfc97bfbe30e759c364c8d1cca058540e471aa5991c2d48bc4f72cd5ad3 WatchSource:0}: Error finding container 72b81dfc97bfbe30e759c364c8d1cca058540e471aa5991c2d48bc4f72cd5ad3: Status 404 returned error can't find the container with id 72b81dfc97bfbe30e759c364c8d1cca058540e471aa5991c2d48bc4f72cd5ad3 Nov 01 15:30:01.215942 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:30:01.212491 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ace21e_49b9_462f_9da9_ca9192b5ea25.slice/crio-5b08900be066d1135a1391de1dbe1a090b7f7da6ad2a6107ab9abd969823b43c WatchSource:0}: Error finding container 5b08900be066d1135a1391de1dbe1a090b7f7da6ad2a6107ab9abd969823b43c: Status 404 returned error can't find the container with id 5b08900be066d1135a1391de1dbe1a090b7f7da6ad2a6107ab9abd969823b43c Nov 01 15:30:01.219509 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.219479 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln"] Nov 01 15:30:01.224713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.224553 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8"] Nov 01 15:30:01.238493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.238471 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt"] Nov 01 15:30:01.285481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.285458 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5"] Nov 01 15:30:01.297403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.297380 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9"] Nov 01 15:30:01.695472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.695382 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" event={"ID":"612b19a9-4cda-4f47-8f2e-cc0af02fdb31","Type":"ContainerStarted","Data":"c313fb9b5e272a6c04abc11e1adb1c9283fee36747719c2316540e49dc6d23d6"} Nov 01 15:30:01.696817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.696790 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" event={"ID":"1fad72ab-7969-4d90-a4b4-5799378fe079","Type":"ContainerStarted","Data":"07e52a9e2546c03087d143632dff70cac39799c8e8840054e3ed2c35601670fe"} Nov 01 15:30:01.698459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.698432 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerStarted","Data":"b49f29f0a9ddbf48cef651dcf1af6232d877f4327ac18d62e25b136088ab2f5f"} Nov 01 15:30:01.698577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.698487 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerStarted","Data":"820cc6531ea5ea4528f5d0dfbea90a6e20473b13b98d371c1be1591c47e88d67"} Nov 01 15:30:01.700034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.700001 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerStarted","Data":"5223ee85b858be765acfc1eeb698ac358e6377cdddf448c133cbf778ef024c48"} Nov 01 15:30:01.700278 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.700048 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerStarted","Data":"c3e15e877db0cdc029b7bd43fd31c64e38a9d1b36c29080ba9aad1994866fa1b"} Nov 01 15:30:01.701833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.701807 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerStarted","Data":"001b9382c8ce903e61d4159db9687e54e415dc678c450168166be101131941b0"} Nov 01 15:30:01.701941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.701847 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerStarted","Data":"4b4a2fdbb2be1783cc984399c21e343ff4fa32e9b7e40e710b0794cd626b89ce"} Nov 01 15:30:01.703133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.703107 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerStarted","Data":"5b08900be066d1135a1391de1dbe1a090b7f7da6ad2a6107ab9abd969823b43c"} Nov 01 15:30:01.704355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.704328 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" event={"ID":"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660","Type":"ContainerStarted","Data":"c42719185c2a8c075f9d1bab076fb38560cced6cd3c6e141ea58941d76225535"} Nov 01 15:30:01.705763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.705736 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerStarted","Data":"72b81dfc97bfbe30e759c364c8d1cca058540e471aa5991c2d48bc4f72cd5ad3"} Nov 01 15:30:01.707265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.707244 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerStarted","Data":"1b917a8b0b9c6dfe91464fd9a5d67a160e8e751353621334c1578d8dd1d079be"} Nov 01 15:30:01.707350 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.707288 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerStarted","Data":"164e9bce0d61a996a02d12c8c24b21f32276d74535831b611fa17ef6a5f997da"} Nov 01 15:30:01.708516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.708497 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerStarted","Data":"f1d8c79cca17ad5a2d42b951b1cba8ab1a7c15be2df4b84389416298874ef34e"} Nov 01 15:30:01.709693 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.709672 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerStarted","Data":"a23aec687f24b5d56b6baa496b9a3b20ea36d42792cd9f9d7f5ab2c2cf8623eb"} Nov 01 15:30:01.813752 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.813697 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" podStartSLOduration=1.8136842990000002 podStartE2EDuration="1.813684299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:01.81047584 +0000 UTC m=+4340.868627669" watchObservedRunningTime="2026-11-01 15:30:01.813684299 +0000 UTC m=+4340.871836180" Nov 01 15:30:01.860305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.860277 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:01.930508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:01.930426 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" podStartSLOduration=1.930405041 podStartE2EDuration="1.930405041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:01.922569727 +0000 UTC m=+4340.980721576" watchObservedRunningTime="2026-11-01 15:30:01.930405041 +0000 UTC m=+4340.988556891" Nov 01 15:30:02.156250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.156176 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" podStartSLOduration=2.156158061 podStartE2EDuration="2.156158061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:02.154426869 +0000 UTC m=+4341.212578718" watchObservedRunningTime="2026-11-01 15:30:02.156158061 +0000 UTC m=+4341.214309933" Nov 01 15:30:02.158497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.158459 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" podStartSLOduration=2.158446865 podStartE2EDuration="2.158446865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:02.042648016 +0000 UTC m=+4341.100799865" watchObservedRunningTime="2026-11-01 15:30:02.158446865 +0000 UTC m=+4341.216598712" Nov 01 15:30:02.719158 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.719130 5473 generic.go:356] "Generic (PLEG): container finished" podID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerID="001b9382c8ce903e61d4159db9687e54e415dc678c450168166be101131941b0" exitCode=60 Nov 01 15:30:02.719299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.719242 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerDied","Data":"001b9382c8ce903e61d4159db9687e54e415dc678c450168166be101131941b0"} Nov 01 15:30:02.719532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.719516 5473 scope.go:122] "RemoveContainer" containerID="001b9382c8ce903e61d4159db9687e54e415dc678c450168166be101131941b0" Nov 01 15:30:02.720982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.720963 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerStarted","Data":"d29a8d63be4520da1f71df378480c5e319706cd354801d2781dca5d2d2456574"} Nov 01 15:30:02.722714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.722695 5473 generic.go:356] "Generic (PLEG): container finished" podID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerID="aa48da8f3d5e124f25623d5cf80574fa32a7bf1c17b4ea04250b8d29e8323269" exitCode=60 Nov 01 15:30:02.722842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.722769 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" event={"ID":"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660","Type":"ContainerDied","Data":"aa48da8f3d5e124f25623d5cf80574fa32a7bf1c17b4ea04250b8d29e8323269"} Nov 01 15:30:02.723085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.722975 5473 scope.go:122] "RemoveContainer" containerID="aa48da8f3d5e124f25623d5cf80574fa32a7bf1c17b4ea04250b8d29e8323269" Nov 01 15:30:02.725119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.725091 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerStarted","Data":"cbe5a43bd505e46a882d8ffd8cd49c9cbc8c43f325c54057a24171813539a9a2"} Nov 01 15:30:02.727121 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.727089 5473 generic.go:356] "Generic (PLEG): container finished" podID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerID="1b917a8b0b9c6dfe91464fd9a5d67a160e8e751353621334c1578d8dd1d079be" exitCode=60 Nov 01 15:30:02.727266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.727197 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerDied","Data":"1b917a8b0b9c6dfe91464fd9a5d67a160e8e751353621334c1578d8dd1d079be"} Nov 01 15:30:02.727538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.727518 5473 scope.go:122] "RemoveContainer" containerID="1b917a8b0b9c6dfe91464fd9a5d67a160e8e751353621334c1578d8dd1d079be" Nov 01 15:30:02.729245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.729191 5473 generic.go:356] "Generic (PLEG): container finished" podID="d6e434e8-3a12-4943-a4fd-579026276228" containerID="a7b3ce71fa4cce18f5c749106834018454eef2fddc08ab4993c94592c1aef477" exitCode=60 Nov 01 15:30:02.729326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.729248 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerDied","Data":"a7b3ce71fa4cce18f5c749106834018454eef2fddc08ab4993c94592c1aef477"} Nov 01 15:30:02.729647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.729628 5473 scope.go:122] "RemoveContainer" containerID="a7b3ce71fa4cce18f5c749106834018454eef2fddc08ab4993c94592c1aef477" Nov 01 15:30:02.733151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.732374 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerStarted","Data":"f87c23fdde49b6d93c5e8d47e6a3ea2760b68e9488fd1bf583029bd2445743fe"} Nov 01 15:30:02.735389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.734915 5473 generic.go:356] "Generic (PLEG): container finished" podID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerID="f7d293fbb07802bd23f83444c7d07fc20b56f5d050a5c43b85e17b832c7046e2" exitCode=60 Nov 01 15:30:02.735389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.734973 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" event={"ID":"612b19a9-4cda-4f47-8f2e-cc0af02fdb31","Type":"ContainerDied","Data":"f7d293fbb07802bd23f83444c7d07fc20b56f5d050a5c43b85e17b832c7046e2"} Nov 01 15:30:02.735389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.735194 5473 scope.go:122] "RemoveContainer" containerID="f7d293fbb07802bd23f83444c7d07fc20b56f5d050a5c43b85e17b832c7046e2" Nov 01 15:30:02.738255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.738230 5473 generic.go:356] "Generic (PLEG): container finished" podID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerID="a5cb1b3b7ff4c4992864251696da0ba0ae0e00800571c03fc69fd0701cfd6743" exitCode=60 Nov 01 15:30:02.738375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.738322 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" event={"ID":"1fad72ab-7969-4d90-a4b4-5799378fe079","Type":"ContainerDied","Data":"a5cb1b3b7ff4c4992864251696da0ba0ae0e00800571c03fc69fd0701cfd6743"} Nov 01 15:30:02.738515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.738498 5473 scope.go:122] "RemoveContainer" containerID="a5cb1b3b7ff4c4992864251696da0ba0ae0e00800571c03fc69fd0701cfd6743" Nov 01 15:30:02.740040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.740020 5473 generic.go:356] "Generic (PLEG): container finished" podID="6d54414d-4f51-4dec-a597-da362f0958ca" containerID="b49f29f0a9ddbf48cef651dcf1af6232d877f4327ac18d62e25b136088ab2f5f" exitCode=60 Nov 01 15:30:02.740177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.740042 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerDied","Data":"b49f29f0a9ddbf48cef651dcf1af6232d877f4327ac18d62e25b136088ab2f5f"} Nov 01 15:30:02.740388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.740369 5473 scope.go:122] "RemoveContainer" containerID="b49f29f0a9ddbf48cef651dcf1af6232d877f4327ac18d62e25b136088ab2f5f" Nov 01 15:30:02.741976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.741957 5473 generic.go:356] "Generic (PLEG): container finished" podID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerID="5223ee85b858be765acfc1eeb698ac358e6377cdddf448c133cbf778ef024c48" exitCode=60 Nov 01 15:30:02.742079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.742000 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerDied","Data":"5223ee85b858be765acfc1eeb698ac358e6377cdddf448c133cbf778ef024c48"} Nov 01 15:30:02.742257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:02.742208 5473 scope.go:122] "RemoveContainer" containerID="5223ee85b858be765acfc1eeb698ac358e6377cdddf448c133cbf778ef024c48" Nov 01 15:30:03.286418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.286363 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" podStartSLOduration=3.286343257 podStartE2EDuration="3.286343257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:03.284754189 +0000 UTC m=+4342.342906039" watchObservedRunningTime="2026-11-01 15:30:03.286343257 +0000 UTC m=+4342.344495109" Nov 01 15:30:03.399471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.399376 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" podStartSLOduration=3.399361553 podStartE2EDuration="3.399361553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:03.396735198 +0000 UTC m=+4342.454887048" watchObservedRunningTime="2026-11-01 15:30:03.399361553 +0000 UTC m=+4342.457513427" Nov 01 15:30:03.743772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.743664 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" podStartSLOduration=3.7436455669999997 podStartE2EDuration="3.743645567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:03.738573765 +0000 UTC m=+4342.796725615" watchObservedRunningTime="2026-11-01 15:30:03.743645567 +0000 UTC m=+4342.801797417" Nov 01 15:30:03.753557 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.753524 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerStarted","Data":"77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf"} Nov 01 15:30:03.755268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.755240 5473 generic.go:356] "Generic (PLEG): container finished" podID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerID="f87c23fdde49b6d93c5e8d47e6a3ea2760b68e9488fd1bf583029bd2445743fe" exitCode=60 Nov 01 15:30:03.755391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.755328 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerDied","Data":"f87c23fdde49b6d93c5e8d47e6a3ea2760b68e9488fd1bf583029bd2445743fe"} Nov 01 15:30:03.755584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.755566 5473 scope.go:122] "RemoveContainer" containerID="f87c23fdde49b6d93c5e8d47e6a3ea2760b68e9488fd1bf583029bd2445743fe" Nov 01 15:30:03.757430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.757407 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" event={"ID":"612b19a9-4cda-4f47-8f2e-cc0af02fdb31","Type":"ContainerStarted","Data":"4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295"} Nov 01 15:30:03.759544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.759515 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" event={"ID":"1fad72ab-7969-4d90-a4b4-5799378fe079","Type":"ContainerStarted","Data":"df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11"} Nov 01 15:30:03.761474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.761449 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerStarted","Data":"eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d"} Nov 01 15:30:03.763385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.763357 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerStarted","Data":"bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa"} Nov 01 15:30:03.765274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.765254 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerStarted","Data":"423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894"} Nov 01 15:30:03.766846 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.766828 5473 generic.go:356] "Generic (PLEG): container finished" podID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerID="d29a8d63be4520da1f71df378480c5e319706cd354801d2781dca5d2d2456574" exitCode=60 Nov 01 15:30:03.766936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.766877 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerDied","Data":"d29a8d63be4520da1f71df378480c5e319706cd354801d2781dca5d2d2456574"} Nov 01 15:30:03.767091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.767074 5473 scope.go:122] "RemoveContainer" containerID="d29a8d63be4520da1f71df378480c5e319706cd354801d2781dca5d2d2456574" Nov 01 15:30:03.769040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.769024 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" event={"ID":"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660","Type":"ContainerStarted","Data":"4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd"} Nov 01 15:30:03.770640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.770615 5473 generic.go:356] "Generic (PLEG): container finished" podID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerID="cbe5a43bd505e46a882d8ffd8cd49c9cbc8c43f325c54057a24171813539a9a2" exitCode=60 Nov 01 15:30:03.770740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.770653 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerDied","Data":"cbe5a43bd505e46a882d8ffd8cd49c9cbc8c43f325c54057a24171813539a9a2"} Nov 01 15:30:03.770893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.770878 5473 scope.go:122] "RemoveContainer" containerID="cbe5a43bd505e46a882d8ffd8cd49c9cbc8c43f325c54057a24171813539a9a2" Nov 01 15:30:03.772745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:03.772722 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerStarted","Data":"1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346"} Nov 01 15:30:04.077860 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.077813 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" podStartSLOduration=4.077797125 podStartE2EDuration="4.077797125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:04.076309187 +0000 UTC m=+4343.134461038" watchObservedRunningTime="2026-11-01 15:30:04.077797125 +0000 UTC m=+4343.135949003" Nov 01 15:30:04.377527 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.377487 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b19a9_4cda_4f47_8f2e_cc0af02fdb31.slice/crio-conmon-4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612b19a9_4cda_4f47_8f2e_cc0af02fdb31.slice/crio-4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bd9d94_00f9_4ee0_88b7_c9bb75b05660.slice/crio-conmon-4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd.scope\": RecentStats: unable to find data in memory cache]" Nov 01 15:30:04.418130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.418079 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" podStartSLOduration=4.418064964 podStartE2EDuration="4.418064964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:04.41461564 +0000 UTC m=+4343.472767489" watchObservedRunningTime="2026-11-01 15:30:04.418064964 +0000 UTC m=+4343.476216814" Nov 01 15:30:04.787248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.787183 5473 generic.go:356] "Generic (PLEG): container finished" podID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerID="4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd" exitCode=60 Nov 01 15:30:04.787452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.787275 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" event={"ID":"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660","Type":"ContainerDied","Data":"4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd"} Nov 01 15:30:04.787452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.787345 5473 scope.go:122] "RemoveContainer" containerID="aa48da8f3d5e124f25623d5cf80574fa32a7bf1c17b4ea04250b8d29e8323269" Nov 01 15:30:04.787602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.787582 5473 scope.go:122] "RemoveContainer" containerID="4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd" Nov 01 15:30:04.788001 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.787974 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5_odh-ai-gateway-infra(e4bd9d94-00f9-4ee0-88b7-c9bb75b05660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" Nov 01 15:30:04.789560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.789534 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerStarted","Data":"2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f"} Nov 01 15:30:04.791472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.791449 5473 generic.go:356] "Generic (PLEG): container finished" podID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerID="1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346" exitCode=60 Nov 01 15:30:04.791552 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.791521 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerDied","Data":"1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346"} Nov 01 15:30:04.791743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.791729 5473 scope.go:122] "RemoveContainer" containerID="1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346" Nov 01 15:30:04.792038 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.792013 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk_odh-ai-gateway-infra(f33ff80d-3e42-4d5e-a8a9-1f1544012b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" Nov 01 15:30:04.793540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.793519 5473 generic.go:356] "Generic (PLEG): container finished" podID="d6e434e8-3a12-4943-a4fd-579026276228" containerID="77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf" exitCode=60 Nov 01 15:30:04.793632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.793567 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerDied","Data":"77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf"} Nov 01 15:30:04.794170 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.793821 5473 scope.go:122] "RemoveContainer" containerID="77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf" Nov 01 15:30:04.794170 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.794131 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g_odh-ai-gateway-infra(d6e434e8-3a12-4943-a4fd-579026276228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" podUID="d6e434e8-3a12-4943-a4fd-579026276228" Nov 01 15:30:04.795607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.795586 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerStarted","Data":"694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358"} Nov 01 15:30:04.797607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.797588 5473 generic.go:356] "Generic (PLEG): container finished" podID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerID="4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295" exitCode=60 Nov 01 15:30:04.797709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.797637 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" event={"ID":"612b19a9-4cda-4f47-8f2e-cc0af02fdb31","Type":"ContainerDied","Data":"4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295"} Nov 01 15:30:04.797930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.797911 5473 scope.go:122] "RemoveContainer" containerID="4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295" Nov 01 15:30:04.798193 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.798168 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9_odh-ai-gateway-infra(612b19a9-4cda-4f47-8f2e-cc0af02fdb31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" Nov 01 15:30:04.799517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.799498 5473 generic.go:356] "Generic (PLEG): container finished" podID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerID="df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11" exitCode=60 Nov 01 15:30:04.799587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.799547 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" event={"ID":"1fad72ab-7969-4d90-a4b4-5799378fe079","Type":"ContainerDied","Data":"df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11"} Nov 01 15:30:04.799818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.799795 5473 scope.go:122] "RemoveContainer" containerID="df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11" Nov 01 15:30:04.800046 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.800024 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt_odh-ai-gateway-infra(1fad72ab-7969-4d90-a4b4-5799378fe079)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" Nov 01 15:30:04.801643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.801622 5473 generic.go:356] "Generic (PLEG): container finished" podID="6d54414d-4f51-4dec-a597-da362f0958ca" containerID="eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d" exitCode=60 Nov 01 15:30:04.801735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.801680 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerDied","Data":"eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d"} Nov 01 15:30:04.801885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.801836 5473 scope.go:122] "RemoveContainer" containerID="1b917a8b0b9c6dfe91464fd9a5d67a160e8e751353621334c1578d8dd1d079be" Nov 01 15:30:04.801943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.801927 5473 scope.go:122] "RemoveContainer" containerID="eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d" Nov 01 15:30:04.802165 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.802137 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9_odh-ai-gateway-infra(6d54414d-4f51-4dec-a597-da362f0958ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" Nov 01 15:30:04.803486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.803464 5473 generic.go:356] "Generic (PLEG): container finished" podID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerID="bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa" exitCode=60 Nov 01 15:30:04.803558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.803538 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerDied","Data":"bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa"} Nov 01 15:30:04.803737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.803721 5473 scope.go:122] "RemoveContainer" containerID="bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa" Nov 01 15:30:04.803955 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.803932 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86_odh-ai-gateway-infra(8e528eda-7ad6-4f05-b88d-a19125f1eacc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" Nov 01 15:30:04.805376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.805355 5473 generic.go:356] "Generic (PLEG): container finished" podID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerID="423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894" exitCode=60 Nov 01 15:30:04.805462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.805448 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerDied","Data":"423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894"} Nov 01 15:30:04.805627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.805606 5473 scope.go:122] "RemoveContainer" containerID="423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894" Nov 01 15:30:04.805863 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:04.805836 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd_odh-ai-gateway-infra(48423bb2-dc91-41b2-9ea5-ac1770ea3312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" Nov 01 15:30:04.807501 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.807481 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerStarted","Data":"97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c"} Nov 01 15:30:04.814774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.814754 5473 scope.go:122] "RemoveContainer" containerID="a7b3ce71fa4cce18f5c749106834018454eef2fddc08ab4993c94592c1aef477" Nov 01 15:30:04.827093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.827067 5473 scope.go:122] "RemoveContainer" containerID="f7d293fbb07802bd23f83444c7d07fc20b56f5d050a5c43b85e17b832c7046e2" Nov 01 15:30:04.839377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.839355 5473 scope.go:122] "RemoveContainer" containerID="a5cb1b3b7ff4c4992864251696da0ba0ae0e00800571c03fc69fd0701cfd6743" Nov 01 15:30:04.851712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.851693 5473 scope.go:122] "RemoveContainer" containerID="b49f29f0a9ddbf48cef651dcf1af6232d877f4327ac18d62e25b136088ab2f5f" Nov 01 15:30:04.864429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.864399 5473 scope.go:122] "RemoveContainer" containerID="5223ee85b858be765acfc1eeb698ac358e6377cdddf448c133cbf778ef024c48" Nov 01 15:30:04.867781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.867734 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" podStartSLOduration=4.867720934 podStartE2EDuration="4.867720934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:04.866372017 +0000 UTC m=+4343.924523867" watchObservedRunningTime="2026-11-01 15:30:04.867720934 +0000 UTC m=+4343.925872813" Nov 01 15:30:04.877871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.877852 5473 scope.go:122] "RemoveContainer" containerID="001b9382c8ce903e61d4159db9687e54e415dc678c450168166be101131941b0" Nov 01 15:30:04.984073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:04.984020 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" podStartSLOduration=4.984000495 podStartE2EDuration="4.984000495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:30:04.981926692 +0000 UTC m=+4344.040078544" watchObservedRunningTime="2026-11-01 15:30:04.984000495 +0000 UTC m=+4344.042152345" Nov 01 15:30:05.817558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.817530 5473 scope.go:122] "RemoveContainer" containerID="4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295" Nov 01 15:30:05.818007 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.817864 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9_odh-ai-gateway-infra(612b19a9-4cda-4f47-8f2e-cc0af02fdb31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" Nov 01 15:30:05.819273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.819252 5473 scope.go:122] "RemoveContainer" containerID="df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11" Nov 01 15:30:05.819544 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.819521 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt_odh-ai-gateway-infra(1fad72ab-7969-4d90-a4b4-5799378fe079)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" Nov 01 15:30:05.820858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.820838 5473 scope.go:122] "RemoveContainer" containerID="eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d" Nov 01 15:30:05.821095 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.821077 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9_odh-ai-gateway-infra(6d54414d-4f51-4dec-a597-da362f0958ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" Nov 01 15:30:05.822415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.822397 5473 scope.go:122] "RemoveContainer" containerID="bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa" Nov 01 15:30:05.822637 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.822618 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86_odh-ai-gateway-infra(8e528eda-7ad6-4f05-b88d-a19125f1eacc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" Nov 01 15:30:05.823991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.823965 5473 scope.go:122] "RemoveContainer" containerID="423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894" Nov 01 15:30:05.824197 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.824178 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd_odh-ai-gateway-infra(48423bb2-dc91-41b2-9ea5-ac1770ea3312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" Nov 01 15:30:05.825517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.825498 5473 generic.go:356] "Generic (PLEG): container finished" podID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerID="97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c" exitCode=60 Nov 01 15:30:05.825602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.825576 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerDied","Data":"97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c"} Nov 01 15:30:05.825665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.825634 5473 scope.go:122] "RemoveContainer" containerID="d29a8d63be4520da1f71df378480c5e319706cd354801d2781dca5d2d2456574" Nov 01 15:30:05.825762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.825744 5473 scope.go:122] "RemoveContainer" containerID="97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c" Nov 01 15:30:05.826032 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.826003 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln_odh-ai-gateway-infra(03ace21e-49b9-462f-9da9-ca9192b5ea25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" Nov 01 15:30:05.831920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.827552 5473 scope.go:122] "RemoveContainer" containerID="4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd" Nov 01 15:30:05.831920 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.827778 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5_odh-ai-gateway-infra(e4bd9d94-00f9-4ee0-88b7-c9bb75b05660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" Nov 01 15:30:05.834017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.833993 5473 generic.go:356] "Generic (PLEG): container finished" podID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerID="2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f" exitCode=60 Nov 01 15:30:05.834129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.834037 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerDied","Data":"2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f"} Nov 01 15:30:05.834477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.834460 5473 scope.go:122] "RemoveContainer" containerID="2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f" Nov 01 15:30:05.834770 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.834746 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl_odh-ai-gateway-infra(eaddc32d-14c2-4d45-947f-ff47b9f154d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" Nov 01 15:30:05.836001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.835985 5473 scope.go:122] "RemoveContainer" containerID="1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346" Nov 01 15:30:05.836262 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.836201 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk_odh-ai-gateway-infra(f33ff80d-3e42-4d5e-a8a9-1f1544012b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" Nov 01 15:30:05.837658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.837641 5473 scope.go:122] "RemoveContainer" containerID="77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf" Nov 01 15:30:05.837861 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.837830 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g_odh-ai-gateway-infra(d6e434e8-3a12-4943-a4fd-579026276228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" podUID="d6e434e8-3a12-4943-a4fd-579026276228" Nov 01 15:30:05.839333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.839311 5473 generic.go:356] "Generic (PLEG): container finished" podID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerID="694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358" exitCode=60 Nov 01 15:30:05.839431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.839367 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerDied","Data":"694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358"} Nov 01 15:30:05.839566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.839554 5473 scope.go:122] "RemoveContainer" containerID="694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358" Nov 01 15:30:05.839739 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:05.839724 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8_odh-ai-gateway-infra(a7fd14ef-9b6d-4846-adc3-6389bec6f367)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" Nov 01 15:30:05.840919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.840902 5473 scope.go:122] "RemoveContainer" containerID="cbe5a43bd505e46a882d8ffd8cd49c9cbc8c43f325c54057a24171813539a9a2" Nov 01 15:30:05.852491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:05.852474 5473 scope.go:122] "RemoveContainer" containerID="f87c23fdde49b6d93c5e8d47e6a3ea2760b68e9488fd1bf583029bd2445743fe" Nov 01 15:30:06.130315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:06.130203 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:06.290848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:06.290819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:06.849249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:06.849204 5473 scope.go:122] "RemoveContainer" containerID="97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c" Nov 01 15:30:06.849725 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:06.849599 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln_odh-ai-gateway-infra(03ace21e-49b9-462f-9da9-ca9192b5ea25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" Nov 01 15:30:06.850906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:06.850883 5473 scope.go:122] "RemoveContainer" containerID="2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f" Nov 01 15:30:06.851209 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:06.851189 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl_odh-ai-gateway-infra(eaddc32d-14c2-4d45-947f-ff47b9f154d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" Nov 01 15:30:06.852609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:06.852590 5473 scope.go:122] "RemoveContainer" containerID="694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358" Nov 01 15:30:06.852865 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:06.852847 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8_odh-ai-gateway-infra(a7fd14ef-9b6d-4846-adc3-6389bec6f367)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" Nov 01 15:30:10.704592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:10.704552 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:15.139062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:15.138995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:16.075239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:16.074897 5473 scope.go:122] "RemoveContainer" containerID="97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c" Nov 01 15:30:16.081237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:16.075849 5473 scope.go:122] "RemoveContainer" containerID="4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295" Nov 01 15:30:17.079240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.076682 5473 scope.go:122] "RemoveContainer" containerID="423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894" Nov 01 15:30:17.079240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.077514 5473 scope.go:122] "RemoveContainer" containerID="bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa" Nov 01 15:30:17.079240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.078183 5473 scope.go:122] "RemoveContainer" containerID="df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11" Nov 01 15:30:17.079240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.078845 5473 scope.go:122] "RemoveContainer" containerID="77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf" Nov 01 15:30:17.134067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.112347 5473 scope.go:122] "RemoveContainer" containerID="4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd" Nov 01 15:30:17.134067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.116155 5473 scope.go:122] "RemoveContainer" containerID="2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f" Nov 01 15:30:17.134067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.118846 5473 scope.go:122] "RemoveContainer" containerID="694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358" Nov 01 15:30:17.348385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:17.348348 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 15:30:18.016316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.016288 5473 generic.go:356] "Generic (PLEG): container finished" podID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerID="c43fa58e0871d0e454866439f3a31ab237a46624032f9dbe026aacaed8998c2a" exitCode=60 Nov 01 15:30:18.016503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.016354 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerDied","Data":"c43fa58e0871d0e454866439f3a31ab237a46624032f9dbe026aacaed8998c2a"} Nov 01 15:30:18.016503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.016411 5473 scope.go:122] "RemoveContainer" containerID="97caf62ce97d7018ba24cdb4f39a69ea118e0c7f0d4a02ab0d5aec8d0d69b22c" Nov 01 15:30:18.016850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.016831 5473 scope.go:122] "RemoveContainer" containerID="c43fa58e0871d0e454866439f3a31ab237a46624032f9dbe026aacaed8998c2a" Nov 01 15:30:18.017205 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:18.017178 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln_odh-ai-gateway-infra(03ace21e-49b9-462f-9da9-ca9192b5ea25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" Nov 01 15:30:18.018830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.018812 5473 generic.go:356] "Generic (PLEG): container finished" podID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerID="1a2dc13f96062d799799b2d5130eb7dc2b3497150ec8d89d91c2ff61e762e87b" exitCode=60 Nov 01 15:30:18.018905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.018883 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" event={"ID":"612b19a9-4cda-4f47-8f2e-cc0af02fdb31","Type":"ContainerDied","Data":"1a2dc13f96062d799799b2d5130eb7dc2b3497150ec8d89d91c2ff61e762e87b"} Nov 01 15:30:18.019262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.019248 5473 scope.go:122] "RemoveContainer" containerID="1a2dc13f96062d799799b2d5130eb7dc2b3497150ec8d89d91c2ff61e762e87b" Nov 01 15:30:18.019545 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:18.019524 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9_odh-ai-gateway-infra(612b19a9-4cda-4f47-8f2e-cc0af02fdb31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" Nov 01 15:30:18.031342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.030753 5473 scope.go:122] "RemoveContainer" containerID="4a9a80547b95e635d4cd46c155ea103a62d859c708af071bcb24b4445bfb6295" Nov 01 15:30:18.075031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:18.075004 5473 scope.go:122] "RemoveContainer" containerID="eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d" Nov 01 15:30:19.029190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.029165 5473 generic.go:356] "Generic (PLEG): container finished" podID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerID="d071924925e1849ed38d0b6edadd76926914592a9de1c40bf7fa9e7da3e29b01" exitCode=60 Nov 01 15:30:19.029571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.029250 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerDied","Data":"d071924925e1849ed38d0b6edadd76926914592a9de1c40bf7fa9e7da3e29b01"} Nov 01 15:30:19.029571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.029374 5473 scope.go:122] "RemoveContainer" containerID="bc16976b463e0365447aa8ed4a40a2bff28ace0a935e8ec597f850c7fdca64fa" Nov 01 15:30:19.031799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.031780 5473 scope.go:122] "RemoveContainer" containerID="d071924925e1849ed38d0b6edadd76926914592a9de1c40bf7fa9e7da3e29b01" Nov 01 15:30:19.032266 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:19.032151 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86_odh-ai-gateway-infra(8e528eda-7ad6-4f05-b88d-a19125f1eacc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" Nov 01 15:30:19.032266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.032247 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerStarted","Data":"f4fae2eee3cf88040e6744bff72142d1124aec19ac8b544af04d8522ab239165"} Nov 01 15:30:19.037114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.037093 5473 generic.go:356] "Generic (PLEG): container finished" podID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerID="ea05852eb6191d1e01e8b46dee5ce692feff06c828f5f5241149a9c853c495b4" exitCode=60 Nov 01 15:30:19.037227 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.037126 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" event={"ID":"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660","Type":"ContainerDied","Data":"ea05852eb6191d1e01e8b46dee5ce692feff06c828f5f5241149a9c853c495b4"} Nov 01 15:30:19.037496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.037480 5473 scope.go:122] "RemoveContainer" containerID="ea05852eb6191d1e01e8b46dee5ce692feff06c828f5f5241149a9c853c495b4" Nov 01 15:30:19.037764 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:19.037746 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5_odh-ai-gateway-infra(e4bd9d94-00f9-4ee0-88b7-c9bb75b05660)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" Nov 01 15:30:19.040140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.040044 5473 generic.go:356] "Generic (PLEG): container finished" podID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerID="9051b76b0b044d23ac1da042343fbdc52af77777d920ff91b111929ed3dea2e7" exitCode=60 Nov 01 15:30:19.040271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.040137 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerDied","Data":"9051b76b0b044d23ac1da042343fbdc52af77777d920ff91b111929ed3dea2e7"} Nov 01 15:30:19.040593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.040572 5473 scope.go:122] "RemoveContainer" containerID="9051b76b0b044d23ac1da042343fbdc52af77777d920ff91b111929ed3dea2e7" Nov 01 15:30:19.040899 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:19.040858 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl_odh-ai-gateway-infra(eaddc32d-14c2-4d45-947f-ff47b9f154d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" Nov 01 15:30:19.046091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.046066 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerStarted","Data":"db7360c60cd3c7c29903c03b2ca642582251ab5de116260c6fee308bab3c021b"} Nov 01 15:30:19.049186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.049023 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerStarted","Data":"df5f069cd422224403c7908d56eb2208946b3eeebe8b19371619fbab6b0c4e36"} Nov 01 15:30:19.049610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.049425 5473 scope.go:122] "RemoveContainer" containerID="df5f069cd422224403c7908d56eb2208946b3eeebe8b19371619fbab6b0c4e36" Nov 01 15:30:19.049716 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:19.049689 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8_odh-ai-gateway-infra(a7fd14ef-9b6d-4846-adc3-6389bec6f367)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" Nov 01 15:30:19.053953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.053931 5473 generic.go:356] "Generic (PLEG): container finished" podID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerID="753068dbda623d87c54ba55341cdb4e1528f581f99939495596061f0a5844b27" exitCode=60 Nov 01 15:30:19.054047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.054002 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" event={"ID":"1fad72ab-7969-4d90-a4b4-5799378fe079","Type":"ContainerDied","Data":"753068dbda623d87c54ba55341cdb4e1528f581f99939495596061f0a5844b27"} Nov 01 15:30:19.054388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.054370 5473 scope.go:122] "RemoveContainer" containerID="753068dbda623d87c54ba55341cdb4e1528f581f99939495596061f0a5844b27" Nov 01 15:30:19.054653 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:19.054628 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt_odh-ai-gateway-infra(1fad72ab-7969-4d90-a4b4-5799378fe079)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" Nov 01 15:30:19.056141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.056118 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerStarted","Data":"1934302e3f667388f166ab105666f9c42452a9611d2256ca93d694544a799d5b"} Nov 01 15:30:19.075127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.075109 5473 scope.go:122] "RemoveContainer" containerID="1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346" Nov 01 15:30:19.143650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.143623 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln"] Nov 01 15:30:19.251029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.250987 5473 scope.go:122] "RemoveContainer" containerID="4da39bb27fb5b170633bc95698eab5cdabe6eb60410c651f246ab2b8ceb9aecd" Nov 01 15:30:19.263330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.263053 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9"] Nov 01 15:30:19.308011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.307967 5473 scope.go:122] "RemoveContainer" containerID="2a00e1237d2f51d9aa1d4f6f7f39a3d5c126656bff21af1ae1f63b4984b2b27f" Nov 01 15:30:19.335149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.332689 5473 scope.go:122] "RemoveContainer" containerID="df3b34d305e35ff0729ce0a81eb117449941b883144b3fd9fb0d71dec9cb2a11" Nov 01 15:30:19.342319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.342296 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:19.447123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.447098 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:19.467348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.467316 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7\" (UniqueName: \"kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7\") pod \"03ace21e-49b9-462f-9da9-ca9192b5ea25\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " Nov 01 15:30:19.467573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.467554 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca\") pod \"03ace21e-49b9-462f-9da9-ca9192b5ea25\" (UID: \"03ace21e-49b9-462f-9da9-ca9192b5ea25\") " Nov 01 15:30:19.467887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.467867 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca" pod "03ace21e-49b9-462f-9da9-ca9192b5ea25" (UID: "03ace21e-49b9-462f-9da9-ca9192b5ea25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:19.468245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.468207 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ace21e-49b9-462f-9da9-ca9192b5ea25-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:19.469358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.469337 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7" pod "03ace21e-49b9-462f-9da9-ca9192b5ea25" (UID: "03ace21e-49b9-462f-9da9-ca9192b5ea25"). InnerVolumeSpecName "kube-api-access-5k9t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:19.569246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.569190 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca\") pod \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " Nov 01 15:30:19.569370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.569268 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8\" (UniqueName: \"kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8\") pod \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\" (UID: \"612b19a9-4cda-4f47-8f2e-cc0af02fdb31\") " Nov 01 15:30:19.569656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.569604 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca" pod "612b19a9-4cda-4f47-8f2e-cc0af02fdb31" (UID: "612b19a9-4cda-4f47-8f2e-cc0af02fdb31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:19.569895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.569878 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:19.569992 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.569905 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k9t7\" (UniqueName: \"kubernetes.io/projected/03ace21e-49b9-462f-9da9-ca9192b5ea25-kube-api-access-5k9t7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:19.571663 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.571638 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8" pod "612b19a9-4cda-4f47-8f2e-cc0af02fdb31" (UID: "612b19a9-4cda-4f47-8f2e-cc0af02fdb31"). InnerVolumeSpecName "kube-api-access-85dc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:19.670804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.670777 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85dc8\" (UniqueName: \"kubernetes.io/projected/612b19a9-4cda-4f47-8f2e-cc0af02fdb31-kube-api-access-85dc8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:19.887056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:19.886957 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:20.069199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.069163 5473 generic.go:356] "Generic (PLEG): container finished" podID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerID="f4fae2eee3cf88040e6744bff72142d1124aec19ac8b544af04d8522ab239165" exitCode=60 Nov 01 15:30:20.069794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.069307 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerDied","Data":"f4fae2eee3cf88040e6744bff72142d1124aec19ac8b544af04d8522ab239165"} Nov 01 15:30:20.069794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.069371 5473 scope.go:122] "RemoveContainer" containerID="423038157a4a5cc43980910b49eb90ac515f1bda17a79781ccad9ebbe174c894" Nov 01 15:30:20.069922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.069796 5473 scope.go:122] "RemoveContainer" containerID="f4fae2eee3cf88040e6744bff72142d1124aec19ac8b544af04d8522ab239165" Nov 01 15:30:20.070193 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:20.070162 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd_odh-ai-gateway-infra(48423bb2-dc91-41b2-9ea5-ac1770ea3312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" Nov 01 15:30:20.072409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.072379 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" event={"ID":"03ace21e-49b9-462f-9da9-ca9192b5ea25","Type":"ContainerDied","Data":"5b08900be066d1135a1391de1dbe1a090b7f7da6ad2a6107ab9abd969823b43c"} Nov 01 15:30:20.072513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.072489 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln" Nov 01 15:30:20.083621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.083595 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerStarted","Data":"c64379bf593b9f23649795218fd6abc850b31a428ec71a09192521ad232e1cf7"} Nov 01 15:30:20.087695 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.087673 5473 scope.go:122] "RemoveContainer" containerID="c43fa58e0871d0e454866439f3a31ab237a46624032f9dbe026aacaed8998c2a" Nov 01 15:30:20.089441 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.089420 5473 generic.go:356] "Generic (PLEG): container finished" podID="d6e434e8-3a12-4943-a4fd-579026276228" containerID="db7360c60cd3c7c29903c03b2ca642582251ab5de116260c6fee308bab3c021b" exitCode=60 Nov 01 15:30:20.089547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.089489 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerDied","Data":"db7360c60cd3c7c29903c03b2ca642582251ab5de116260c6fee308bab3c021b"} Nov 01 15:30:20.089871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.089853 5473 scope.go:122] "RemoveContainer" containerID="db7360c60cd3c7c29903c03b2ca642582251ab5de116260c6fee308bab3c021b" Nov 01 15:30:20.090130 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:20.090107 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g_odh-ai-gateway-infra(d6e434e8-3a12-4943-a4fd-579026276228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" podUID="d6e434e8-3a12-4943-a4fd-579026276228" Nov 01 15:30:20.091629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.091604 5473 generic.go:356] "Generic (PLEG): container finished" podID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerID="df5f069cd422224403c7908d56eb2208946b3eeebe8b19371619fbab6b0c4e36" exitCode=60 Nov 01 15:30:20.091727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.091630 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerDied","Data":"df5f069cd422224403c7908d56eb2208946b3eeebe8b19371619fbab6b0c4e36"} Nov 01 15:30:20.091962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.091946 5473 scope.go:122] "RemoveContainer" containerID="df5f069cd422224403c7908d56eb2208946b3eeebe8b19371619fbab6b0c4e36" Nov 01 15:30:20.092260 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:20.092203 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8_odh-ai-gateway-infra(a7fd14ef-9b6d-4846-adc3-6389bec6f367)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" Nov 01 15:30:20.093288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.093268 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" Nov 01 15:30:20.093364 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.093288 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9" event={"ID":"612b19a9-4cda-4f47-8f2e-cc0af02fdb31","Type":"ContainerDied","Data":"c313fb9b5e272a6c04abc11e1adb1c9283fee36747719c2316540e49dc6d23d6"} Nov 01 15:30:20.097516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.097498 5473 generic.go:356] "Generic (PLEG): container finished" podID="6d54414d-4f51-4dec-a597-da362f0958ca" containerID="1934302e3f667388f166ab105666f9c42452a9611d2256ca93d694544a799d5b" exitCode=60 Nov 01 15:30:20.097608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.097526 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerDied","Data":"1934302e3f667388f166ab105666f9c42452a9611d2256ca93d694544a799d5b"} Nov 01 15:30:20.097798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.097784 5473 scope.go:122] "RemoveContainer" containerID="1934302e3f667388f166ab105666f9c42452a9611d2256ca93d694544a799d5b" Nov 01 15:30:20.097995 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:20.097978 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9_odh-ai-gateway-infra(6d54414d-4f51-4dec-a597-da362f0958ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" Nov 01 15:30:20.099432 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.099416 5473 scope.go:122] "RemoveContainer" containerID="77dc616af24f50aa195654a137f3a70527e20757ac3a7c1bf92db5d3c3daeeaf" Nov 01 15:30:20.117664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.117641 5473 scope.go:122] "RemoveContainer" containerID="694b3f01f5879bd7bb696c8adad6689da0e6672b73d2f32a22237aeba1869358" Nov 01 15:30:20.132285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.132201 5473 scope.go:122] "RemoveContainer" containerID="1a2dc13f96062d799799b2d5130eb7dc2b3497150ec8d89d91c2ff61e762e87b" Nov 01 15:30:20.158746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.158717 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86"] Nov 01 15:30:20.158963 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.158935 5473 scope.go:122] "RemoveContainer" containerID="eeae473aa19ef0686e98513220544f39e2430248e5aac018161ebcb6b653051d" Nov 01 15:30:20.271555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.271524 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g"] Nov 01 15:30:20.343060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.343038 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:20.386265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.386211 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8"] Nov 01 15:30:20.483168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.483085 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca\") pod \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " Nov 01 15:30:20.483370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.483184 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7\" (UniqueName: \"kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7\") pod \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\" (UID: \"8e528eda-7ad6-4f05-b88d-a19125f1eacc\") " Nov 01 15:30:20.483492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.483464 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca" pod "8e528eda-7ad6-4f05-b88d-a19125f1eacc" (UID: "8e528eda-7ad6-4f05-b88d-a19125f1eacc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:20.483795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.483774 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e528eda-7ad6-4f05-b88d-a19125f1eacc-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:20.485402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.485379 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7" pod "8e528eda-7ad6-4f05-b88d-a19125f1eacc" (UID: "8e528eda-7ad6-4f05-b88d-a19125f1eacc"). InnerVolumeSpecName "kube-api-access-kzmm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:20.499707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.499682 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl"] Nov 01 15:30:20.585804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.585770 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmm7\" (UniqueName: \"kubernetes.io/projected/8e528eda-7ad6-4f05-b88d-a19125f1eacc-kube-api-access-kzmm7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:20.619919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.619888 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt"] Nov 01 15:30:20.666148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.666123 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:20.730643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.730607 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd"] Nov 01 15:30:20.788284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.788252 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr\" (UniqueName: \"kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr\") pod \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " Nov 01 15:30:20.788641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.788617 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca\") pod \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\" (UID: \"eaddc32d-14c2-4d45-947f-ff47b9f154d3\") " Nov 01 15:30:20.789052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.789024 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca" pod "eaddc32d-14c2-4d45-947f-ff47b9f154d3" (UID: "eaddc32d-14c2-4d45-947f-ff47b9f154d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:20.789236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.789201 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaddc32d-14c2-4d45-947f-ff47b9f154d3-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:20.790416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.790390 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr" pod "eaddc32d-14c2-4d45-947f-ff47b9f154d3" (UID: "eaddc32d-14c2-4d45-947f-ff47b9f154d3"). InnerVolumeSpecName "kube-api-access-f2ssr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:20.808152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.808131 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:20.842437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.842415 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9"] Nov 01 15:30:20.889712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.889691 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca\") pod \"1fad72ab-7969-4d90-a4b4-5799378fe079\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " Nov 01 15:30:20.889827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.889734 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k\" (UniqueName: \"kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k\") pod \"1fad72ab-7969-4d90-a4b4-5799378fe079\" (UID: \"1fad72ab-7969-4d90-a4b4-5799378fe079\") " Nov 01 15:30:20.890054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.890034 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca" pod "1fad72ab-7969-4d90-a4b4-5799378fe079" (UID: "1fad72ab-7969-4d90-a4b4-5799378fe079"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:20.890417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.890392 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fad72ab-7969-4d90-a4b4-5799378fe079-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:20.890508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.890421 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2ssr\" (UniqueName: \"kubernetes.io/projected/eaddc32d-14c2-4d45-947f-ff47b9f154d3-kube-api-access-f2ssr\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:20.891848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.891822 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k" pod "1fad72ab-7969-4d90-a4b4-5799378fe079" (UID: "1fad72ab-7969-4d90-a4b4-5799378fe079"). InnerVolumeSpecName "kube-api-access-8nj4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:20.964004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.963966 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5"] Nov 01 15:30:20.985516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.985489 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln"] Nov 01 15:30:20.991939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.991913 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nj4k\" (UniqueName: \"kubernetes.io/projected/1fad72ab-7969-4d90-a4b4-5799378fe079-kube-api-access-8nj4k\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:20.991939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:20.991926 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-96165a-29892450-5ldln"] Nov 01 15:30:21.088198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.088167 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" path="/var/lib/kubelet/pods/03ace21e-49b9-462f-9da9-ca9192b5ea25/volumes" Nov 01 15:30:21.095779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.095751 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:21.115131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.115091 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" Nov 01 15:30:21.115276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.115161 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt" event={"ID":"1fad72ab-7969-4d90-a4b4-5799378fe079","Type":"ContainerDied","Data":"07e52a9e2546c03087d143632dff70cac39799c8e8840054e3ed2c35601670fe"} Nov 01 15:30:21.115276 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.115259 5473 scope.go:122] "RemoveContainer" containerID="753068dbda623d87c54ba55341cdb4e1528f581f99939495596061f0a5844b27" Nov 01 15:30:21.122236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.120145 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" event={"ID":"8e528eda-7ad6-4f05-b88d-a19125f1eacc","Type":"ContainerDied","Data":"c3e15e877db0cdc029b7bd43fd31c64e38a9d1b36c29080ba9aad1994866fa1b"} Nov 01 15:30:21.122236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.120955 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86" Nov 01 15:30:21.122236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.121739 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9"] Nov 01 15:30:21.136383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.136346 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" event={"ID":"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660","Type":"ContainerDied","Data":"c42719185c2a8c075f9d1bab076fb38560cced6cd3c6e141ea58941d76225535"} Nov 01 15:30:21.136517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.136459 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5" Nov 01 15:30:21.138248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.136717 5473 scope.go:122] "RemoveContainer" containerID="d071924925e1849ed38d0b6edadd76926914592a9de1c40bf7fa9e7da3e29b01" Nov 01 15:30:21.143443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.143290 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" event={"ID":"eaddc32d-14c2-4d45-947f-ff47b9f154d3","Type":"ContainerDied","Data":"72b81dfc97bfbe30e759c364c8d1cca058540e471aa5991c2d48bc4f72cd5ad3"} Nov 01 15:30:21.144271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.143633 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl" Nov 01 15:30:21.149783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.149757 5473 generic.go:356] "Generic (PLEG): container finished" podID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerID="c64379bf593b9f23649795218fd6abc850b31a428ec71a09192521ad232e1cf7" exitCode=60 Nov 01 15:30:21.149865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.149819 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerDied","Data":"c64379bf593b9f23649795218fd6abc850b31a428ec71a09192521ad232e1cf7"} Nov 01 15:30:21.154117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.154092 5473 scope.go:122] "RemoveContainer" containerID="c64379bf593b9f23649795218fd6abc850b31a428ec71a09192521ad232e1cf7" Nov 01 15:30:21.154502 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:21.154472 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk_odh-ai-gateway-infra(f33ff80d-3e42-4d5e-a8a9-1f1544012b30)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" Nov 01 15:30:21.157806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.157660 5473 scope.go:122] "RemoveContainer" containerID="ea05852eb6191d1e01e8b46dee5ce692feff06c828f5f5241149a9c853c495b4" Nov 01 15:30:21.162104 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.161796 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b3f06c-29892450-wj4t9"] Nov 01 15:30:21.172168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.172089 5473 scope.go:122] "RemoveContainer" containerID="9051b76b0b044d23ac1da042343fbdc52af77777d920ff91b111929ed3dea2e7" Nov 01 15:30:21.184605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.184336 5473 scope.go:122] "RemoveContainer" containerID="1f9ca4e1719f6f9bb9557e4e8698a5028f549139c3304ee58f3d8df4723dc346" Nov 01 15:30:21.193948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.193830 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca\") pod \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " Nov 01 15:30:21.193948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.193939 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt\") pod \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\" (UID: \"e4bd9d94-00f9-4ee0-88b7-c9bb75b05660\") " Nov 01 15:30:21.194492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.194466 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca" pod "e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" (UID: "e4bd9d94-00f9-4ee0-88b7-c9bb75b05660"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:21.194755 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.194735 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.203299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.202626 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt" pod "e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" (UID: "e4bd9d94-00f9-4ee0-88b7-c9bb75b05660"). InnerVolumeSpecName "kube-api-access-5q8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:21.293441 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.293415 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:21.296502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.296476 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q8rt\" (UniqueName: \"kubernetes.io/projected/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660-kube-api-access-5q8rt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.307358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.307328 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk"] Nov 01 15:30:21.332590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.332565 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86"] Nov 01 15:30:21.340467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.340436 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-00a787-29892450-8dq86"] Nov 01 15:30:21.352801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.352776 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:21.357385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.357366 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:21.397164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397121 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca\") pod \"d6e434e8-3a12-4943-a4fd-579026276228\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " Nov 01 15:30:21.397328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397174 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca\") pod \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " Nov 01 15:30:21.397426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397366 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2\" (UniqueName: \"kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2\") pod \"6d54414d-4f51-4dec-a597-da362f0958ca\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " Nov 01 15:30:21.397426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397402 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97\" (UniqueName: \"kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97\") pod \"d6e434e8-3a12-4943-a4fd-579026276228\" (UID: \"d6e434e8-3a12-4943-a4fd-579026276228\") " Nov 01 15:30:21.397651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397480 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv\" (UniqueName: \"kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv\") pod \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\" (UID: \"a7fd14ef-9b6d-4846-adc3-6389bec6f367\") " Nov 01 15:30:21.397651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397540 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca" pod "d6e434e8-3a12-4943-a4fd-579026276228" (UID: "d6e434e8-3a12-4943-a4fd-579026276228"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:21.397651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397557 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca\") pod \"6d54414d-4f51-4dec-a597-da362f0958ca\" (UID: \"6d54414d-4f51-4dec-a597-da362f0958ca\") " Nov 01 15:30:21.397968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.397936 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca" pod "a7fd14ef-9b6d-4846-adc3-6389bec6f367" (UID: "a7fd14ef-9b6d-4846-adc3-6389bec6f367"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:21.398306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.398282 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6e434e8-3a12-4943-a4fd-579026276228-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.398410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.398311 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd14ef-9b6d-4846-adc3-6389bec6f367-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.398410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.398308 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca" pod "6d54414d-4f51-4dec-a597-da362f0958ca" (UID: "6d54414d-4f51-4dec-a597-da362f0958ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:21.400543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.400397 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97" pod "d6e434e8-3a12-4943-a4fd-579026276228" (UID: "d6e434e8-3a12-4943-a4fd-579026276228"). InnerVolumeSpecName "kube-api-access-rtl97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:21.400543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.400432 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv" pod "a7fd14ef-9b6d-4846-adc3-6389bec6f367" (UID: "a7fd14ef-9b6d-4846-adc3-6389bec6f367"). InnerVolumeSpecName "kube-api-access-6lwdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:21.400543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.400456 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2" pod "6d54414d-4f51-4dec-a597-da362f0958ca" (UID: "6d54414d-4f51-4dec-a597-da362f0958ca"). InnerVolumeSpecName "kube-api-access-mlgl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:21.438061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.438040 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:21.499350 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.499323 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca\") pod \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " Nov 01 15:30:21.499641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.499622 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t\" (UniqueName: \"kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t\") pod \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\" (UID: \"48423bb2-dc91-41b2-9ea5-ac1770ea3312\") " Nov 01 15:30:21.499855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.499833 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca" pod "48423bb2-dc91-41b2-9ea5-ac1770ea3312" (UID: "48423bb2-dc91-41b2-9ea5-ac1770ea3312"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:21.500138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.500119 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwdv\" (UniqueName: \"kubernetes.io/projected/a7fd14ef-9b6d-4846-adc3-6389bec6f367-kube-api-access-6lwdv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.500138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.500137 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48423bb2-dc91-41b2-9ea5-ac1770ea3312-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.500304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.500149 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d54414d-4f51-4dec-a597-da362f0958ca-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.500304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.500158 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlgl2\" (UniqueName: \"kubernetes.io/projected/6d54414d-4f51-4dec-a597-da362f0958ca-kube-api-access-mlgl2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.500304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.500167 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtl97\" (UniqueName: \"kubernetes.io/projected/d6e434e8-3a12-4943-a4fd-579026276228-kube-api-access-rtl97\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.501500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.501480 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t" pod "48423bb2-dc91-41b2-9ea5-ac1770ea3312" (UID: "48423bb2-dc91-41b2-9ea5-ac1770ea3312"). InnerVolumeSpecName "kube-api-access-b2v7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:21.601717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.601694 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2v7t\" (UniqueName: \"kubernetes.io/projected/48423bb2-dc91-41b2-9ea5-ac1770ea3312-kube-api-access-b2v7t\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:21.750051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.750019 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl"] Nov 01 15:30:21.756300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.756265 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-52fffa-29892450-h4fwl"] Nov 01 15:30:21.922988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.922901 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt"] Nov 01 15:30:21.926527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:21.926501 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1f8e2c-29892450-mfwlt"] Nov 01 15:30:22.160459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.160421 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" event={"ID":"6d54414d-4f51-4dec-a597-da362f0958ca","Type":"ContainerDied","Data":"820cc6531ea5ea4528f5d0dfbea90a6e20473b13b98d371c1be1591c47e88d67"} Nov 01 15:30:22.160944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.160487 5473 scope.go:122] "RemoveContainer" containerID="1934302e3f667388f166ab105666f9c42452a9611d2256ca93d694544a799d5b" Nov 01 15:30:22.160944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.160428 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9" Nov 01 15:30:22.163272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.163251 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" Nov 01 15:30:22.163403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.163247 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd" event={"ID":"48423bb2-dc91-41b2-9ea5-ac1770ea3312","Type":"ContainerDied","Data":"4b4a2fdbb2be1783cc984399c21e343ff4fa32e9b7e40e710b0794cd626b89ce"} Nov 01 15:30:22.168702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.168679 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" Nov 01 15:30:22.168805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.168718 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g" event={"ID":"d6e434e8-3a12-4943-a4fd-579026276228","Type":"ContainerDied","Data":"f1d8c79cca17ad5a2d42b951b1cba8ab1a7c15be2df4b84389416298874ef34e"} Nov 01 15:30:22.171558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.171536 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" Nov 01 15:30:22.171664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.171538 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8" event={"ID":"a7fd14ef-9b6d-4846-adc3-6389bec6f367","Type":"ContainerDied","Data":"a23aec687f24b5d56b6baa496b9a3b20ea36d42792cd9f9d7f5ab2c2cf8623eb"} Nov 01 15:30:22.174168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.174142 5473 scope.go:122] "RemoveContainer" containerID="f4fae2eee3cf88040e6744bff72142d1124aec19ac8b544af04d8522ab239165" Nov 01 15:30:22.185901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.185883 5473 scope.go:122] "RemoveContainer" containerID="db7360c60cd3c7c29903c03b2ca642582251ab5de116260c6fee308bab3c021b" Nov 01 15:30:22.195266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.195245 5473 scope.go:122] "RemoveContainer" containerID="df5f069cd422224403c7908d56eb2208946b3eeebe8b19371619fbab6b0c4e36" Nov 01 15:30:22.354624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.354600 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:22.411453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.411426 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca\") pod \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " Nov 01 15:30:22.411631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.411616 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk\" (UniqueName: \"kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk\") pod \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\" (UID: \"f33ff80d-3e42-4d5e-a8a9-1f1544012b30\") " Nov 01 15:30:22.411808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.411781 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca" pod "f33ff80d-3e42-4d5e-a8a9-1f1544012b30" (UID: "f33ff80d-3e42-4d5e-a8a9-1f1544012b30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:30:22.412066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.412048 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:22.413718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.413695 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk" pod "f33ff80d-3e42-4d5e-a8a9-1f1544012b30" (UID: "f33ff80d-3e42-4d5e-a8a9-1f1544012b30"). InnerVolumeSpecName "kube-api-access-95qhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:30:22.513302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.513269 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95qhk\" (UniqueName: \"kubernetes.io/projected/f33ff80d-3e42-4d5e-a8a9-1f1544012b30-kube-api-access-95qhk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:30:22.555880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.555848 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g"] Nov 01 15:30:22.559599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.559571 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-55943a-29892450-ncd6g"] Nov 01 15:30:22.728507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.728430 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8"] Nov 01 15:30:22.737899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.737871 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4e07cb-29892450-2gcc8"] Nov 01 15:30:22.910075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.910043 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd"] Nov 01 15:30:22.913548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:22.913519 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-007270-29892450-x4xxd"] Nov 01 15:30:23.083966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.083936 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" path="/var/lib/kubelet/pods/1fad72ab-7969-4d90-a4b4-5799378fe079/volumes" Nov 01 15:30:23.084428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.084411 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" path="/var/lib/kubelet/pods/48423bb2-dc91-41b2-9ea5-ac1770ea3312/volumes" Nov 01 15:30:23.084918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.084897 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" path="/var/lib/kubelet/pods/612b19a9-4cda-4f47-8f2e-cc0af02fdb31/volumes" Nov 01 15:30:23.085713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.085694 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" path="/var/lib/kubelet/pods/8e528eda-7ad6-4f05-b88d-a19125f1eacc/volumes" Nov 01 15:30:23.086250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.086226 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" path="/var/lib/kubelet/pods/a7fd14ef-9b6d-4846-adc3-6389bec6f367/volumes" Nov 01 15:30:23.086812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.086784 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e434e8-3a12-4943-a4fd-579026276228" path="/var/lib/kubelet/pods/d6e434e8-3a12-4943-a4fd-579026276228/volumes" Nov 01 15:30:23.087596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.087578 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" path="/var/lib/kubelet/pods/eaddc32d-14c2-4d45-947f-ff47b9f154d3/volumes" Nov 01 15:30:23.088050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.088035 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9"] Nov 01 15:30:23.094832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.094811 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1ac11c-29892450-rhjf9"] Nov 01 15:30:23.181646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.181617 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" Nov 01 15:30:23.181646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.181632 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk" event={"ID":"f33ff80d-3e42-4d5e-a8a9-1f1544012b30","Type":"ContainerDied","Data":"164e9bce0d61a996a02d12c8c24b21f32276d74535831b611fa17ef6a5f997da"} Nov 01 15:30:23.182124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.181693 5473 scope.go:122] "RemoveContainer" containerID="c64379bf593b9f23649795218fd6abc850b31a428ec71a09192521ad232e1cf7" Nov 01 15:30:23.206970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.206865 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5"] Nov 01 15:30:23.210706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.210686 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-af2dd2-29892450-j4ph5"] Nov 01 15:30:23.382101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.382006 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk"] Nov 01 15:30:23.385606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:23.385580 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-97c87c-29892450-qqlfk"] Nov 01 15:30:24.321206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:24.321175 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:25.083425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:25.083384 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" path="/var/lib/kubelet/pods/6d54414d-4f51-4dec-a597-da362f0958ca/volumes" Nov 01 15:30:25.083954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:25.083928 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" path="/var/lib/kubelet/pods/e4bd9d94-00f9-4ee0-88b7-c9bb75b05660/volumes" Nov 01 15:30:25.084478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:25.084451 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" path="/var/lib/kubelet/pods/f33ff80d-3e42-4d5e-a8a9-1f1544012b30/volumes" Nov 01 15:30:28.700711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:28.700670 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:31.108457 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:30:31.108422 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:30:33.113464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:33.113431 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:37.464138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:37.464103 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:40.593996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:40.593960 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:40.770294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:40.770257 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:42.008408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:42.008373 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:46.419825 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:46.419795 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:46.580192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:46.580158 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:47.011093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:47.011056 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:47.426146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:47.426095 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:48.751629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:48.751589 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:49.001234 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:49.001198 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:51.222848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:51.222816 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:51.397660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:51.397627 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:52.491769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:52.491735 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:52.658668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:52.658634 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:52.865893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:52.865852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:53.214989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:53.214495 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:53.402715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:53.402671 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:55.974518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:55.974486 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:56.148826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:56.148790 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:57.605628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:57.605594 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:30:57.796060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:30:57.796029 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:00.936642 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:00.936606 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:01.109187 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:31:01.109144 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:31:02.710181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:02.710149 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:02.871857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:02.871779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:05.396928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:05.395371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:06.421572 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:06.421539 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:08.243118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243081 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg"] Nov 01 15:31:08.243943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243919 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243944 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243965 5473 cpu_manager.go:414] "containerMap: removing container" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243973 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243985 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.243992 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244017 5473 cpu_manager.go:414] "containerMap: removing container" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244024 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244038 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244044 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244056 5473 cpu_manager.go:414] "containerMap: removing container" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244063 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244072 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244079 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244088 5473 cpu_manager.go:414] "containerMap: removing container" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244096 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244108 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244116 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244125 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244134 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244143 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244150 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244158 5473 cpu_manager.go:414] "containerMap: removing container" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244164 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244177 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244185 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244194 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244201 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244210 5473 cpu_manager.go:414] "containerMap: removing container" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244232 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244257 5473 cpu_manager.go:414] "containerMap: removing container" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244264 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244275 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244281 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244292 5473 cpu_manager.go:414] "containerMap: removing container" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244298 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244309 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244318 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244338 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244346 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244355 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244363 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244404 5473 cpu_manager.go:414] "containerMap: removing container" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244414 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244427 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244435 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244445 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244452 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244462 5473 cpu_manager.go:414] "containerMap: removing container" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244470 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244673 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244686 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244697 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244707 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244718 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244726 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244735 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244745 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244755 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244764 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244771 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244780 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244789 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244798 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244807 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244815 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244823 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244835 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244843 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ace21e-49b9-462f-9da9-ca9192b5ea25" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244852 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244862 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244871 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244881 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:31:08.244855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244889 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:31:08.247631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244902 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:31:08.247631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244911 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:31:08.247631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244920 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:31:08.247631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244929 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:31:08.247631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.244936 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:31:08.249244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.249195 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.312127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.312100 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg"] Nov 01 15:31:08.336173 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.336149 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.336327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.336246 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf8fl\" (UniqueName: \"kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.336381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.336336 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.348798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.348776 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:31:08.348896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.348873 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:31:08.437728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.437701 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf8fl\" (UniqueName: \"kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.437888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.437736 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.437947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.437888 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.438558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.438533 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.440143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.440124 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.494567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.494508 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf8fl\" (UniqueName: \"kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl\") pod \"maas-api-revoke-sub-62f7d669-zdfvg\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.563508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.563483 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:08.944500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:08.944470 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg"] Nov 01 15:31:08.945388 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:31:08.945350 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0a156b_8563_4413_9191_c4a4cdc963c9.slice/crio-806d8f0d0da932c61d181571915ee522fdebcbd08d8eeaae8433d04dfb84a344 WatchSource:0}: Error finding container 806d8f0d0da932c61d181571915ee522fdebcbd08d8eeaae8433d04dfb84a344: Status 404 returned error can't find the container with id 806d8f0d0da932c61d181571915ee522fdebcbd08d8eeaae8433d04dfb84a344 Nov 01 15:31:09.565646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:09.565593 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" event={"ID":"bb0a156b-8563-4413-9191-c4a4cdc963c9","Type":"ContainerStarted","Data":"13d82e5934d39aebb6156d4b4d42d0626f249082776fdab411e01225225953eb"} Nov 01 15:31:09.566123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:09.565656 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" event={"ID":"bb0a156b-8563-4413-9191-c4a4cdc963c9","Type":"ContainerStarted","Data":"806d8f0d0da932c61d181571915ee522fdebcbd08d8eeaae8433d04dfb84a344"} Nov 01 15:31:09.714419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:09.714070 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" podStartSLOduration=1.714055009 podStartE2EDuration="1.714055009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:31:09.7118794 +0000 UTC m=+4408.770031275" watchObservedRunningTime="2026-11-01 15:31:09.714055009 +0000 UTC m=+4408.772206861" Nov 01 15:31:09.913079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:09.913047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:10.574307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:10.574275 5473 generic.go:356] "Generic (PLEG): container finished" podID="bb0a156b-8563-4413-9191-c4a4cdc963c9" containerID="13d82e5934d39aebb6156d4b4d42d0626f249082776fdab411e01225225953eb" exitCode=0 Nov 01 15:31:10.574692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:10.574359 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" event={"ID":"bb0a156b-8563-4413-9191-c4a4cdc963c9","Type":"ContainerDied","Data":"13d82e5934d39aebb6156d4b4d42d0626f249082776fdab411e01225225953eb"} Nov 01 15:31:11.726400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.726374 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:11.880120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.880047 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca\") pod \"bb0a156b-8563-4413-9191-c4a4cdc963c9\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " Nov 01 15:31:11.880279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.880159 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl\" (UniqueName: \"kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl\") pod \"bb0a156b-8563-4413-9191-c4a4cdc963c9\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " Nov 01 15:31:11.880279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.880259 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca\") pod \"bb0a156b-8563-4413-9191-c4a4cdc963c9\" (UID: \"bb0a156b-8563-4413-9191-c4a4cdc963c9\") " Nov 01 15:31:11.880445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.880422 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca" pod "bb0a156b-8563-4413-9191-c4a4cdc963c9" (UID: "bb0a156b-8563-4413-9191-c4a4cdc963c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:31:11.880714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.880692 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:31:11.882278 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.882248 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca" pod "bb0a156b-8563-4413-9191-c4a4cdc963c9" (UID: "bb0a156b-8563-4413-9191-c4a4cdc963c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:31:11.882278 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.882200 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl" pod "bb0a156b-8563-4413-9191-c4a4cdc963c9" (UID: "bb0a156b-8563-4413-9191-c4a4cdc963c9"). InnerVolumeSpecName "kube-api-access-jf8fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:31:11.982362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.982337 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf8fl\" (UniqueName: \"kubernetes.io/projected/bb0a156b-8563-4413-9191-c4a4cdc963c9-kube-api-access-jf8fl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:31:11.982362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:11.982360 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0a156b-8563-4413-9191-c4a4cdc963c9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:31:12.589990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:12.589953 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" event={"ID":"bb0a156b-8563-4413-9191-c4a4cdc963c9","Type":"ContainerDied","Data":"806d8f0d0da932c61d181571915ee522fdebcbd08d8eeaae8433d04dfb84a344"} Nov 01 15:31:12.590168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:12.589993 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg" Nov 01 15:31:12.590168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:12.590004 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="806d8f0d0da932c61d181571915ee522fdebcbd08d8eeaae8433d04dfb84a344" Nov 01 15:31:13.750606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:13.750569 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:13.927754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:13.927720 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:14.446596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:14.446559 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:14.628787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:14.628749 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:18.900760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:18.900723 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:19.089951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:19.089912 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:19.782502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:19.782462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:23.292908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:23.292815 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:25.050823 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:25.050792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:27.673538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:27.673500 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:31.117129 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:31:31.117092 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:31:32.236478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:32.236444 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:36.609486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:36.609454 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:40.688516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:40.688481 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:40.967006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:40.966903 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:45.395951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:45.395912 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:46.602241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:46.590509 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:49.846434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:49.846396 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:53.863369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:53.863333 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:54.274571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:54.274541 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:58.704025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:58.703986 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:31:59.959079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:31:59.959045 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:01.110615 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:32:01.110572 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:32:01.644789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:01.644755 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:01.822041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:01.822004 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:03.038082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:03.038049 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:06.743895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:06.743864 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:07.299547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:07.299509 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:07.459109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:07.459067 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:07.625699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:07.625622 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:11.880630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:11.880601 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:16.219741 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:16.219709 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:16.408863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:16.408828 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:16.572608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:16.572573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:20.909370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:20.909337 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:25.437106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:25.437073 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:28.048068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:28.048030 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh"] Nov 01 15:32:28.097597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:28.097562 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fe9cfcd-rckqh"] Nov 01 15:32:29.082996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:29.082967 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c97256e-a9cf-4bc7-8d45-f199bbc97566" path="/var/lib/kubelet/pods/9c97256e-a9cf-4bc7-8d45-f199bbc97566/volumes" Nov 01 15:32:29.925426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:29.925395 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:31.111180 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:32:31.111147 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:32:34.269316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:34.269287 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:38.623746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:38.623717 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:43.019739 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:43.019701 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:44.613792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:44.613764 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:32:44.634643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:44.634621 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:32:47.407975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:47.407934 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:49.706481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:49.706447 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:50.294262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:50.291967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:50.518962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:50.518930 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:51.822851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:51.822814 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:54.576183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:54.576123 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:54.775423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:54.775392 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:56.540062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:56.540023 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:57.707039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:57.707005 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:32:57.923866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:32:57.923834 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:01.115991 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:33:01.115954 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:33:01.362135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:01.362098 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:05.751351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:05.751315 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:07.063297 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:07.063255 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:10.168658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:10.168626 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:14.366066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:14.366035 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:14.605705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:14.605672 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:19.015875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:19.015842 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:23.413809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:23.413774 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:27.810935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:27.810905 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:31.109613 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:33:31.109576 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:33:32.234962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:32.234922 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:35.087529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.087495 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz"] Nov 01 15:33:35.087944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.087683 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" containerID="cri-o://3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927" gracePeriod=30 Nov 01 15:33:35.097435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.097411 5473 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-f8fzz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.60:8080/ready\": dial tcp 10.134.1.60:8080: connect: connection refused" start-of-body= Nov 01 15:33:35.097546 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.097460 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" probeResult="failure" output="Get \"http://10.134.1.60:8080/ready\": dial tcp 10.134.1.60:8080: connect: connection refused" Nov 01 15:33:35.365616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.365587 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:33:35.462273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462240 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462274 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462326 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462368 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462456 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462533 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462601 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs\") pod \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\" (UID: \"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9\") " Nov 01 15:33:35.462821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.462795 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:33:35.463406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.463381 5473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-ca-configmap\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.465111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.465077 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:33:35.465299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.465139 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:33:35.465390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.465330 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:33:35.465563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.465540 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:33:35.465641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.465563 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:33:35.465703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.465690 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7" pod "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" (UID: "d6e831cf-4fa8-4ec0-8365-8b9c250a9af9"). InnerVolumeSpecName "kube-api-access-llwk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:33:35.565036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.565003 5473 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-cacerts\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.565036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.565032 5473 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-kubeconfig\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.565036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.565042 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.565298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.565051 5473 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-istio-csr-dns-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.565298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.565061 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llwk7\" (UniqueName: \"kubernetes.io/projected/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-kube-api-access-llwk7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.565298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.565071 5473 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9-local-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:33:35.782257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.782203 5473 generic.go:356] "Generic (PLEG): container finished" podID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerID="3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927" exitCode=0 Nov 01 15:33:35.782436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.782265 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" Nov 01 15:33:35.782436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.782283 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" event={"ID":"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9","Type":"ContainerDied","Data":"3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927"} Nov 01 15:33:35.782436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.782338 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz" event={"ID":"d6e831cf-4fa8-4ec0-8365-8b9c250a9af9","Type":"ContainerDied","Data":"91a69b127d66eedef9cc5987a5bf80d21ac726cdf6e8c2947949177c37477644"} Nov 01 15:33:35.782436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.782358 5473 scope.go:122] "RemoveContainer" containerID="3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927" Nov 01 15:33:35.794032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.794010 5473 scope.go:122] "RemoveContainer" containerID="3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927" Nov 01 15:33:35.794298 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:33:35.794278 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927\": container with ID starting with 3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927 not found: ID does not exist" containerID="3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927" Nov 01 15:33:35.794386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.794308 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927"} err="failed to get container status \"3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927\": rpc error: code = NotFound desc = could not find container \"3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927\": container with ID starting with 3db4a647a2e6deca001c5d2ca3de0ca0532cc8aaad69745b236322922b868927 not found: ID does not exist" Nov 01 15:33:35.962518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.962480 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz"] Nov 01 15:33:35.971769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:35.971735 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-f8fzz"] Nov 01 15:33:36.704810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:36.704778 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:37.083024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:37.082990 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" path="/var/lib/kubelet/pods/d6e831cf-4fa8-4ec0-8365-8b9c250a9af9/volumes" Nov 01 15:33:41.011359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:41.011322 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:41.284420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:41.284392 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:44.550324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:44.550286 5473 scope.go:122] "RemoveContainer" containerID="bde400045c3a78b77710611256517a90593a749dea7295bfe3a727c28d20f4be" Nov 01 15:33:44.560939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:44.560919 5473 scope.go:122] "RemoveContainer" containerID="1dfd350677a22d5a2d216604642e12dc6bdd99be44f41910ae4931ef6d41e1b7" Nov 01 15:33:44.570199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:44.570177 5473 scope.go:122] "RemoveContainer" containerID="7913ceac2900d5d7509236e0c07e7c7caeaa7fd215daa5debf47ddce655c0d6c" Nov 01 15:33:45.760644 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:45.760602 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:50.400650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:50.400616 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:51.141926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:51.141893 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:51.558852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:51.558818 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:51.724548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:51.724512 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:52.027224 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027180 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2"] Nov 01 15:33:52.027829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027814 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:33:52.027875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027831 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e528eda-7ad6-4f05-b88d-a19125f1eacc" containerName="cleanup" Nov 01 15:33:52.027875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027846 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:33:52.027875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027851 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d54414d-4f51-4dec-a597-da362f0958ca" containerName="cleanup" Nov 01 15:33:52.027875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027867 5473 cpu_manager.go:414] "containerMap: removing container" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:33:52.027875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027873 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612b19a9-4cda-4f47-8f2e-cc0af02fdb31" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027883 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027888 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e434e8-3a12-4943-a4fd-579026276228" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027900 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bb0a156b-8563-4413-9191-c4a4cdc963c9" containerName="revoke-keys" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027904 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0a156b-8563-4413-9191-c4a4cdc963c9" containerName="revoke-keys" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027919 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027923 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bd9d94-00f9-4ee0-88b7-c9bb75b05660" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027930 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027934 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027950 5473 cpu_manager.go:414] "containerMap: removing container" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027955 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027963 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027967 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027975 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:33:52.028056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.027979 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fd14ef-9b6d-4846-adc3-6389bec6f367" containerName="cleanup" Nov 01 15:33:52.028537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.028074 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33ff80d-3e42-4d5e-a8a9-1f1544012b30" containerName="cleanup" Nov 01 15:33:52.028537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.028082 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="48423bb2-dc91-41b2-9ea5-ac1770ea3312" containerName="cleanup" Nov 01 15:33:52.028537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.028094 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaddc32d-14c2-4d45-947f-ff47b9f154d3" containerName="cleanup" Nov 01 15:33:52.028537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.028101 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0a156b-8563-4413-9191-c4a4cdc963c9" containerName="revoke-keys" Nov 01 15:33:52.028537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.028107 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fad72ab-7969-4d90-a4b4-5799378fe079" containerName="cleanup" Nov 01 15:33:52.028537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.028112 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e831cf-4fa8-4ec0-8365-8b9c250a9af9" containerName="discovery" Nov 01 15:33:52.031471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.031455 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.081091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.081043 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.091442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.091408 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgxvn\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.091728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.091707 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.091902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.091887 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.092059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.092040 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.092201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.092186 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.103238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.092436 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.103781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.103758 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.104017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.103989 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.104335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.097581 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-974865-openshift-default-dockercfg-j2rx4\"" Nov 01 15:33:52.154941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.148419 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:52.162696 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.159082 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2"] Nov 01 15:33:52.205837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.205804 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.205837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.205839 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgxvn\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.205860 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.205920 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.205939 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.205969 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206243 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206291 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206255 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206323 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206368 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206382 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206456 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206560 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.206983 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.206959 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.208879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.208846 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.208989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.208955 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.263701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.263674 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.263884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.263860 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgxvn\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn\") pod \"e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.346659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.346574 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:52.458593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.458556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:52.776494 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.776459 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2"] Nov 01 15:33:52.776986 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:33:52.776961 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a028562_7f7b_4d29_8856_f191a1d6a30d.slice/crio-6bbcd24f6f7ec67a1a878e87008dda1e8c34a0f0c04f6b6c0cba0471dd2b18ba WatchSource:0}: Error finding container 6bbcd24f6f7ec67a1a878e87008dda1e8c34a0f0c04f6b6c0cba0471dd2b18ba: Status 404 returned error can't find the container with id 6bbcd24f6f7ec67a1a878e87008dda1e8c34a0f0c04f6b6c0cba0471dd2b18ba Nov 01 15:33:52.779857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.779820 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:33:52.779954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.779902 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:33:52.779954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.779949 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:33:52.929306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.929264 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" event={"ID":"3a028562-7f7b-4d29-8856-f191a1d6a30d","Type":"ContainerStarted","Data":"173b617f94c566fb08e5d055833b40bf86969f708185e760cbc38474ce607e9f"} Nov 01 15:33:52.929460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:52.929336 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" event={"ID":"3a028562-7f7b-4d29-8856-f191a1d6a30d","Type":"ContainerStarted","Data":"6bbcd24f6f7ec67a1a878e87008dda1e8c34a0f0c04f6b6c0cba0471dd2b18ba"} Nov 01 15:33:53.051986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:53.051885 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podStartSLOduration=2.05187219 podStartE2EDuration="2.05187219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:33:53.048032339 +0000 UTC m=+4572.106184188" watchObservedRunningTime="2026-11-01 15:33:53.05187219 +0000 UTC m=+4572.110024039" Nov 01 15:33:53.347785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:53.347713 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:53.353538 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:53.353509 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:53.936451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:53.936416 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:53.937426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:53.937406 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:33:55.054141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:55.054109 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:55.894513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:55.894481 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:56.073652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:56.073610 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:58.843166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:58.843128 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:59.000857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.000814 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:33:59.208150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.208062 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5"] Nov 01 15:33:59.212637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.212614 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.269935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.269892 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5"] Nov 01 15:33:59.314541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.314499 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-974865\"" Nov 01 15:33:59.379845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.315671 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.387193 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.375746 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-974865-dockercfg-ww456\"" Nov 01 15:33:59.390231 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.390169 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5"] Nov 01 15:33:59.395298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.380209 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vrn\" (UniqueName: \"kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn\") pod \"payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.396067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.396047 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.435755 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.435723 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5"] Nov 01 15:33:59.497086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.497014 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vrn\" (UniqueName: \"kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn\") pod \"payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.497283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.497120 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.497283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.497188 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkm8m\" (UniqueName: \"kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m\") pod \"payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.497405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.497310 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.498025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.498005 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.553820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.553785 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vrn\" (UniqueName: \"kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn\") pod \"payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.598112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.598085 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.598313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.598182 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkm8m\" (UniqueName: \"kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m\") pod \"payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.598845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.598825 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.655044 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.655013 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkm8m\" (UniqueName: \"kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m\") pod \"payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.720697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.720654 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:33:59.827586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.827548 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:33:59.982803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:33:59.982774 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:00.101469 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:34:00.101433 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75dd3c8_9357_48da_9daf_72f409718749.slice/crio-62a1bca3a9d534a70450c5ddf6c21d1e834b5b4ba6c24e5f7c1130ae8a202265 WatchSource:0}: Error finding container 62a1bca3a9d534a70450c5ddf6c21d1e834b5b4ba6c24e5f7c1130ae8a202265: Status 404 returned error can't find the container with id 62a1bca3a9d534a70450c5ddf6c21d1e834b5b4ba6c24e5f7c1130ae8a202265 Nov 01 15:34:00.102042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.102014 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5"] Nov 01 15:34:00.212610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.212585 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5"] Nov 01 15:34:00.212891 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:34:00.212869 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bbdd6a_7154_4bc5_915a_874519683280.slice/crio-f84afddc64ce4bc14f53234df3084864bee9c19fee3988590b986feec043c11f WatchSource:0}: Error finding container f84afddc64ce4bc14f53234df3084864bee9c19fee3988590b986feec043c11f: Status 404 returned error can't find the container with id f84afddc64ce4bc14f53234df3084864bee9c19fee3988590b986feec043c11f Nov 01 15:34:00.993902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.993861 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" event={"ID":"c75dd3c8-9357-48da-9daf-72f409718749","Type":"ContainerStarted","Data":"1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e"} Nov 01 15:34:00.994436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.993949 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" event={"ID":"c75dd3c8-9357-48da-9daf-72f409718749","Type":"ContainerStarted","Data":"62a1bca3a9d534a70450c5ddf6c21d1e834b5b4ba6c24e5f7c1130ae8a202265"} Nov 01 15:34:00.994436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.993973 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:34:00.995684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.995652 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" event={"ID":"e1bbdd6a-7154-4bc5-915a-874519683280","Type":"ContainerStarted","Data":"f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1"} Nov 01 15:34:00.995817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.995707 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" event={"ID":"e1bbdd6a-7154-4bc5-915a-874519683280","Type":"ContainerStarted","Data":"f84afddc64ce4bc14f53234df3084864bee9c19fee3988590b986feec043c11f"} Nov 01 15:34:00.995817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:00.995746 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:34:01.111603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:01.108620 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" podStartSLOduration=2.108604729 podStartE2EDuration="2.108604729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:34:01.107045363 +0000 UTC m=+4580.165197215" watchObservedRunningTime="2026-11-01 15:34:01.108604729 +0000 UTC m=+4580.166756579" Nov 01 15:34:01.111603 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:34:01.110685 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:34:01.225619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:01.224882 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" podStartSLOduration=2.224863276 podStartE2EDuration="2.224863276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:34:01.220167536 +0000 UTC m=+4580.278319387" watchObservedRunningTime="2026-11-01 15:34:01.224863276 +0000 UTC m=+4580.283015127" Nov 01 15:34:04.407944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:04.407889 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:07.393507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:07.393468 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:08.799427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:08.799389 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:12.006241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:12.006185 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:34:12.006631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:12.006285 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:34:13.183759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:13.183729 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:17.127604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.127575 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7"] Nov 01 15:34:17.132405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.132388 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.190056 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.190019 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7"] Nov 01 15:34:17.216760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.216730 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"kube-root-ca.crt\"" Nov 01 15:34:17.219621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.219591 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.219773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.219698 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.219840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.219778 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.219928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.219905 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.220011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.219946 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.220073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.220025 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b4bp\" (UniqueName: \"kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.233333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.233308 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"openshift-service-ca.crt\"" Nov 01 15:34:17.233466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.233343 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"default-dockercfg-tzv99\"" Nov 01 15:34:17.240315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.240297 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ac11c\"/\"rate-test-model-1ac11c-kserve-self-signed-certs\"" Nov 01 15:34:17.255769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.255747 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:17.322082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322045 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b4bp\" (UniqueName: \"kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.322315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322152 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.322315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322289 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.322505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322382 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.322505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322455 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.322505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322486 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.323033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.322996 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.323436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.323036 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.323436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.323084 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.330567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.330511 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.331275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.331251 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.379390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.379306 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b4bp\" (UniqueName: \"kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp\") pod \"rate-test-model-1ac11c-kserve-556c8bd789-4gwx7\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.446047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.445999 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:17.627240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.626145 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:17.830688 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:34:17.830656 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c88bb0b_4801_42ae_9006_6c1006a06c35.slice/crio-aad81375903544426966d73477d342ad2a72aee997ea61a064ced127e450b897 WatchSource:0}: Error finding container aad81375903544426966d73477d342ad2a72aee997ea61a064ced127e450b897: Status 404 returned error can't find the container with id aad81375903544426966d73477d342ad2a72aee997ea61a064ced127e450b897 Nov 01 15:34:17.830837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:17.830712 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7"] Nov 01 15:34:18.126515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:18.126428 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" event={"ID":"4c88bb0b-4801-42ae-9006-6c1006a06c35","Type":"ContainerStarted","Data":"7198851b84cb62df0eac7de66febadac38a6e5088e6d374cdfaf8da295a39b26"} Nov 01 15:34:18.126515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:18.126472 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" event={"ID":"4c88bb0b-4801-42ae-9006-6c1006a06c35","Type":"ContainerStarted","Data":"aad81375903544426966d73477d342ad2a72aee997ea61a064ced127e450b897"} Nov 01 15:34:18.241141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:18.241089 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" podStartSLOduration=1.24107384 podStartE2EDuration="1.24107384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:34:18.239761464 +0000 UTC m=+4597.297913316" watchObservedRunningTime="2026-11-01 15:34:18.24107384 +0000 UTC m=+4597.299225704" Nov 01 15:34:18.446888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:18.446813 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:21.852635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:21.852494 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:22.020935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:22.020902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:22.190874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:22.190777 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:26.407094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:26.407011 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:27.446698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:27.446664 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:27.460295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:27.460270 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:28.216405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:28.216375 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:34:30.052529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:30.052493 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk"] Nov 01 15:34:30.102541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:30.102499 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febbfd88-98rqk"] Nov 01 15:34:30.806066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:30.806036 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:31.084017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:31.083938 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302e2c9b-8141-4828-a92c-e55de67e40c0" path="/var/lib/kubelet/pods/302e2c9b-8141-4828-a92c-e55de67e40c0/volumes" Nov 01 15:34:31.111062 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:34:31.111027 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:34:32.114474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.114443 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd"] Nov 01 15:34:32.119804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.119787 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.186824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.182325 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd"] Nov 01 15:34:32.220272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.220239 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-96165a\"/\"kube-root-ca.crt\"" Nov 01 15:34:32.220272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.220258 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-96165a\"/\"openshift-service-ca.crt\"" Nov 01 15:34:32.221808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.221785 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:32.222035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.222019 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-96165a\"/\"default-dockercfg-grz2j\"" Nov 01 15:34:32.222108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.222023 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-96165a\"/\"test-model-96165a-kserve-self-signed-certs\"" Nov 01 15:34:32.227927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.227893 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.228011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.227978 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.228070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.228040 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qpx\" (UniqueName: \"kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.228199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.228173 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.228279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.228269 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.228437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.228420 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.329892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.329852 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.329900 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330029 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330131 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330180 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330262 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qpx\" (UniqueName: \"kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330296 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330472 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.330562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.330541 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.332296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.332264 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.332553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.332536 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.386970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.386905 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qpx\" (UniqueName: \"kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx\") pod \"test-model-96165a-kserve-8fc6c468c-n4pzd\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.433402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.433375 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:32.811749 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:34:32.811720 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e9ccfa6_26bc_42e7_82da_7a599002ddd2.slice/crio-63625e9c33e2712954eb1475c47842ab1be65d51e2b50e68972e4c365a33cf42 WatchSource:0}: Error finding container 63625e9c33e2712954eb1475c47842ab1be65d51e2b50e68972e4c365a33cf42: Status 404 returned error can't find the container with id 63625e9c33e2712954eb1475c47842ab1be65d51e2b50e68972e4c365a33cf42 Nov 01 15:34:32.811978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:32.811956 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd"] Nov 01 15:34:33.243414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:33.243331 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" event={"ID":"7e9ccfa6-26bc-42e7-82da-7a599002ddd2","Type":"ContainerStarted","Data":"929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305"} Nov 01 15:34:33.243414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:33.243378 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" event={"ID":"7e9ccfa6-26bc-42e7-82da-7a599002ddd2","Type":"ContainerStarted","Data":"63625e9c33e2712954eb1475c47842ab1be65d51e2b50e68972e4c365a33cf42"} Nov 01 15:34:33.359130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:33.359079 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" podStartSLOduration=1.359065159 podStartE2EDuration="1.359065159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:34:33.355552708 +0000 UTC m=+4612.413704557" watchObservedRunningTime="2026-11-01 15:34:33.359065159 +0000 UTC m=+4612.417217080" Nov 01 15:34:33.434263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:33.434234 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:35.227617 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:35.227551 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:36.050204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:36.050163 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp"] Nov 01 15:34:36.100763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:36.100712 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbda1dd4-f74sp"] Nov 01 15:34:37.091498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:37.091457 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12304c5-900d-41f8-8aa2-9cc39f9da462" path="/var/lib/kubelet/pods/c12304c5-900d-41f8-8aa2-9cc39f9da462/volumes" Nov 01 15:34:39.625735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:39.625702 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:42.433573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:42.433540 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:42.445958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:42.445933 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:43.363387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:43.363356 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:34:44.077654 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:44.077614 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:48.504887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:48.504841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:50.453757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.453724 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8"] Nov 01 15:34:50.459886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.459868 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.515977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.515940 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8"] Nov 01 15:34:50.558161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.558121 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.558355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.558272 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.558411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.558367 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.558588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.558572 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.558670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.558620 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.558720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.558701 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rsz\" (UniqueName: \"kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.559919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.559898 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3f45c3\"/\"openshift-service-ca.crt\"" Nov 01 15:34:50.559987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.559934 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3f45c3\"/\"kube-root-ca.crt\"" Nov 01 15:34:50.560037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.560019 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3f45c3\"/\"default-dockercfg-mdx48\"" Nov 01 15:34:50.564692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.564652 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3f45c3\"/\"test-model-3f45c3-kserve-self-signed-certs\"" Nov 01 15:34:50.564934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.564901 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:50.661109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661065 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661237 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661287 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rsz\" (UniqueName: \"kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661431 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661483 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661524 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661556 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661589 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.661891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.661870 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.664318 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.664293 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.664496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.664299 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.718655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.718590 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rsz\" (UniqueName: \"kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz\") pod \"test-model-3f45c3-kserve-f7bd7f68c-2pzp8\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:50.773486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:50.773463 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:51.150852 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:34:51.150813 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce984260_255c_4072_b151_93e42303e8c0.slice/crio-0ba83364e85a7567a0673c43ab4c2271531b2992825e6c6bc635ac6b6c6726f5 WatchSource:0}: Error finding container 0ba83364e85a7567a0673c43ab4c2271531b2992825e6c6bc635ac6b6c6726f5: Status 404 returned error can't find the container with id 0ba83364e85a7567a0673c43ab4c2271531b2992825e6c6bc635ac6b6c6726f5 Nov 01 15:34:51.150967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:51.150919 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8"] Nov 01 15:34:51.422333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:51.422250 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" event={"ID":"ce984260-255c-4072-b151-93e42303e8c0","Type":"ContainerStarted","Data":"0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10"} Nov 01 15:34:51.422333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:51.422294 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" event={"ID":"ce984260-255c-4072-b151-93e42303e8c0","Type":"ContainerStarted","Data":"0ba83364e85a7567a0673c43ab4c2271531b2992825e6c6bc635ac6b6c6726f5"} Nov 01 15:34:51.537805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:51.537751 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" podStartSLOduration=1.537681166 podStartE2EDuration="1.537681166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:34:51.535423673 +0000 UTC m=+4630.593575524" watchObservedRunningTime="2026-11-01 15:34:51.537681166 +0000 UTC m=+4630.595833051" Nov 01 15:34:51.773549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:51.773516 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:34:52.985871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:52.985837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:34:57.339174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:34:57.339136 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:00.593682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:00.593630 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:00.774278 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:00.774243 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:35:00.787159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:00.787131 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:35:01.109050 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:35:01.109012 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:35:01.507198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:01.507124 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:35:01.760747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:01.760647 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:06.180374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:06.180338 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:07.711171 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:07.711133 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:10.912991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:10.912958 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:15.289713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:15.289677 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:19.680625 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:19.680594 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:23.911168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:23.911108 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:24.074025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:24.073988 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:24.235974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:24.235893 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:28.501387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:28.501357 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:31.134754 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:35:31.134713 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:35:32.929734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:32.929702 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:37.365841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:37.365792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:39.081481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:39.081439 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:41.696623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:41.696546 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:44.631016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:44.630984 5473 scope.go:122] "RemoveContainer" containerID="b64f041dce5590fd475d1183298b8c5b46169861c4417c85225ae59fd142be6b" Nov 01 15:35:44.645506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:44.645485 5473 scope.go:122] "RemoveContainer" containerID="e4d2dae2055bf8aa4aa3ee75357caa7ebe332099ff40ded93564e00ef6065a48" Nov 01 15:35:46.074114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:46.074077 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:48.154899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:48.154858 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:50.370268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:50.370165 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:54.713257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:54.713197 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:58.726713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:58.726678 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:35:59.518769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:35:59.518732 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:01.110469 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:36:01.110416 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:36:03.758725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:03.758681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:04.673330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:04.673298 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:06.183065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:06.183033 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:06.340856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:06.340823 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:08.125422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:08.125384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:08.279294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:08.279233 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:10.107738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:10.107705 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:12.048145 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:12.048105 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg"] Nov 01 15:36:12.097647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:12.097620 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62f7d669-zdfvg"] Nov 01 15:36:12.504257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:12.504156 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:13.083002 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:13.082969 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0a156b-8563-4413-9191-c4a4cdc963c9" path="/var/lib/kubelet/pods/bb0a156b-8563-4413-9191-c4a4cdc963c9/volumes" Nov 01 15:36:16.912035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:16.911999 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:21.386321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:21.386286 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:25.783973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:25.783942 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:30.055516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:30.055480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:31.110008 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:36:31.109972 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:36:34.420922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:34.420891 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:38.803827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:38.803791 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:43.108037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:43.108006 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:43.260758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:43.260687 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:47.469835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:47.469806 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:51.859635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:51.859603 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:56.313713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:56.313677 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:36:59.396512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:36:59.396476 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:00.847127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:00.847088 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:01.109059 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:37:01.108971 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:37:05.101994 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:05.101958 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:05.438423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:05.435648 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:06.084978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:06.084946 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:06.269082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:06.269045 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:08.323902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:08.323868 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:09.387890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.387773 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": read tcp 10.134.0.2:59456->10.134.1.75:15021: read: connection reset by peer" start-of-body= Nov 01 15:37:09.387890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.387849 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": read tcp 10.134.0.2:59456->10.134.1.75:15021: read: connection reset by peer" Nov 01 15:37:09.526732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.526699 5473 generic.go:356] "Generic (PLEG): container finished" podID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerID="173b617f94c566fb08e5d055833b40bf86969f708185e760cbc38474ce607e9f" exitCode=0 Nov 01 15:37:09.527207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.526775 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" event={"ID":"3a028562-7f7b-4d29-8856-f191a1d6a30d","Type":"ContainerDied","Data":"173b617f94c566fb08e5d055833b40bf86969f708185e760cbc38474ce607e9f"} Nov 01 15:37:09.527397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.527378 5473 scope.go:122] "RemoveContainer" containerID="173b617f94c566fb08e5d055833b40bf86969f708185e760cbc38474ce607e9f" Nov 01 15:37:09.529318 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.529292 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:37:09.529404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.529370 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:37:09.529444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:09.529406 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:37:10.001092 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:10.001032 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:10.557166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:10.557129 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" event={"ID":"3a028562-7f7b-4d29-8856-f191a1d6a30d","Type":"ContainerStarted","Data":"19cd2acaddd379e738c40831104d738bb5a6011d27306134acb1a62b4ec81750"} Nov 01 15:37:10.886673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:10.886573 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:11.347320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:11.347281 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:37:11.349430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:11.349389 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 15:37:11.349624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:11.349459 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 15:37:12.349604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:12.349568 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 15:37:12.350005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:12.349635 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 15:37:13.348599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.348558 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 15:37:13.348923 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.348894 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 15:37:13.780058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.780020 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw"] Nov 01 15:37:13.780528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.780252 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" containerID="cri-o://3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a" gracePeriod=30 Nov 01 15:37:13.856142 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.856111 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4"] Nov 01 15:37:13.856373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.856330 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" containerID="cri-o://0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a" gracePeriod=30 Nov 01 15:37:13.860672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.860643 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n"] Nov 01 15:37:13.860849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.860817 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" containerID="cri-o://90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1" gracePeriod=30 Nov 01 15:37:13.907347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.907316 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw"] Nov 01 15:37:13.907744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:13.907717 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" containerID="cri-o://3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39" gracePeriod=30 Nov 01 15:37:14.350591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:14.350551 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 15:37:14.350906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:14.350874 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 15:37:14.375671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:14.375635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:15.200422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.200380 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-2zlpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:37:15.200816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.200446 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:37:15.246240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.245848 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-h9b2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:37:15.246240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.245913 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:37:15.253365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.253335 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-stknw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:37:15.253515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.253388 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:37:15.349244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.348547 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 01 15:37:15.349244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.348605 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 01 15:37:15.453906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.453819 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lbfq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:37:15.453906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:15.453883 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:37:16.351251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:16.351206 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:37:16.351725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:16.351713 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:37:16.352564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:16.352540 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:37:18.689816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:18.689781 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:19.140227 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.140189 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:37:19.217978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.217908 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:37:19.232415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232385 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.232561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232445 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.232561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232532 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.232679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232648 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.232870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232769 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.232870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232817 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.232985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.232937 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.233036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.233026 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.233129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.233113 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert\") pod \"ea4c58e2-dded-4c0a-82d7-23240410d460\" (UID: \"ea4c58e2-dded-4c0a-82d7-23240410d460\") " Nov 01 15:37:19.236821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.236443 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:37:19.237293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.237254 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.237635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.237615 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.237800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.237784 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.237957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.237939 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.238717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.238679 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.239513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.239488 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.241413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.241372 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:37:19.242268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.242230 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx" pod "ea4c58e2-dded-4c0a-82d7-23240410d460" (UID: "ea4c58e2-dded-4c0a-82d7-23240410d460"). InnerVolumeSpecName "kube-api-access-sczjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.270386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.270354 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:37:19.278057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.277936 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:37:19.334298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334265 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.334492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334362 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.334492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334442 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.334658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334558 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.334658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334599 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.334774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334680 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.334774 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334699 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.334897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334873 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.334994 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.334964 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.335155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.335049 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.335155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.335070 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.335155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.335128 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.335370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.335198 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy\") pod \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\" (UID: \"c919f6f7-a12e-4fa4-a6d8-8b0adc060037\") " Nov 01 15:37:19.335594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.335556 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:37:19.336050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336014 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336056 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336074 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336088 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336131 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336146 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336161 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336211 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336246 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczjx\" (UniqueName: \"kubernetes.io/projected/ea4c58e2-dded-4c0a-82d7-23240410d460-kube-api-access-sczjx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336261 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336276 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4c58e2-dded-4c0a-82d7-23240410d460-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336290 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336320 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.336387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.336333 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4c58e2-dded-4c0a-82d7-23240410d460-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.337109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.337083 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.337563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.337541 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "kube-api-access-h6rb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.337776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.337754 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.344372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.344347 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo" pod "c919f6f7-a12e-4fa4-a6d8-8b0adc060037" (UID: "c919f6f7-a12e-4fa4-a6d8-8b0adc060037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:37:19.437712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437682 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.437856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437743 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.437856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437765 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.437856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437785 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.437856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437807 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.437856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437829 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.438162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437868 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.438162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.437966 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.438162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438092 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.438162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438127 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.438162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438153 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438144 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438181 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438201 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438255 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438315 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438382 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token\") pod \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\" (UID: \"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0\") " Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438455 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438459 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438502 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438588 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.438609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438612 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438668 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket\") pod \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\" (UID: \"fb5c87e3-80a0-4c1b-827a-a7d78da38a99\") " Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.438994 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439152 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439293 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439354 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439487 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439499 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439521 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439536 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439551 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439577 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439598 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439619 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rb6\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-kube-api-access-h6rb6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439633 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439648 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439661 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439675 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439688 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.439763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.439701 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c919f6f7-a12e-4fa4-a6d8-8b0adc060037-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.440965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.440585 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.440965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.440646 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "kube-api-access-fq724". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.440965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.440651 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:37:19.440965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.440814 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:37:19.441725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.441702 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:37:19.441725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.441717 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.441864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.441746 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz" pod "fb5c87e3-80a0-4c1b-827a-a7d78da38a99" (UID: "fb5c87e3-80a0-4c1b-827a-a7d78da38a99"). InnerVolumeSpecName "kube-api-access-6hkkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.442187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.442164 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token" pod "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" (UID: "520cc1ca-a6c0-47ee-adf4-e288b0fb52d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:37:19.541199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541168 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541199 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541243 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq724\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-kube-api-access-fq724\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541259 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541271 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541284 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541297 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541310 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkkz\" (UniqueName: \"kubernetes.io/projected/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-kube-api-access-6hkkz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.541434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.541323 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb5c87e3-80a0-4c1b-827a-a7d78da38a99-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:37:19.628690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.628652 5473 generic.go:356] "Generic (PLEG): container finished" podID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerID="3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39" exitCode=0 Nov 01 15:37:19.628896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.628697 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" Nov 01 15:37:19.628896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.628729 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" event={"ID":"fb5c87e3-80a0-4c1b-827a-a7d78da38a99","Type":"ContainerDied","Data":"3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39"} Nov 01 15:37:19.628896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.628813 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw" event={"ID":"fb5c87e3-80a0-4c1b-827a-a7d78da38a99","Type":"ContainerDied","Data":"e79f3a9b4c2167e50060f504cb8a44fa94fa11bd9515742447c77b812c217200"} Nov 01 15:37:19.628896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.628842 5473 scope.go:122] "RemoveContainer" containerID="3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39" Nov 01 15:37:19.630526 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.630499 5473 generic.go:356] "Generic (PLEG): container finished" podID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerID="0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a" exitCode=0 Nov 01 15:37:19.630725 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.630706 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" event={"ID":"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0","Type":"ContainerDied","Data":"0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a"} Nov 01 15:37:19.630802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.630754 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" event={"ID":"520cc1ca-a6c0-47ee-adf4-e288b0fb52d0","Type":"ContainerDied","Data":"478231e2032f93bd9bf6e88ad63b590ab14c5d27a0c9357c216df78b713e53c9"} Nov 01 15:37:19.630862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.630823 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4" Nov 01 15:37:19.633181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.633157 5473 generic.go:356] "Generic (PLEG): container finished" podID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerID="3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a" exitCode=0 Nov 01 15:37:19.633339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.633285 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" event={"ID":"ea4c58e2-dded-4c0a-82d7-23240410d460","Type":"ContainerDied","Data":"3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a"} Nov 01 15:37:19.633339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.633326 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" event={"ID":"ea4c58e2-dded-4c0a-82d7-23240410d460","Type":"ContainerDied","Data":"cf6ce9c029d680a66646386b4fc19a3b7457098e9f3b68e90979fee39599f4c6"} Nov 01 15:37:19.633521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.633412 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw" Nov 01 15:37:19.638073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.638048 5473 generic.go:356] "Generic (PLEG): container finished" podID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerID="90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1" exitCode=0 Nov 01 15:37:19.638253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.638122 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" event={"ID":"c919f6f7-a12e-4fa4-a6d8-8b0adc060037","Type":"ContainerDied","Data":"90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1"} Nov 01 15:37:19.638347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.638296 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" event={"ID":"c919f6f7-a12e-4fa4-a6d8-8b0adc060037","Type":"ContainerDied","Data":"dd56e3fb7fb339cfbe083bce40214de44ac2d51ede8cf6bae5d2d367d016e7c2"} Nov 01 15:37:19.638414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.638168 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n" Nov 01 15:37:19.646794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.646775 5473 scope.go:122] "RemoveContainer" containerID="3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39" Nov 01 15:37:19.647186 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:37:19.647160 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39\": container with ID starting with 3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39 not found: ID does not exist" containerID="3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39" Nov 01 15:37:19.647324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.647196 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39"} err="failed to get container status \"3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39\": rpc error: code = NotFound desc = could not find container \"3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39\": container with ID starting with 3e6b49bfc6bd7c5f9340e59a04ecad16608cdbf9494f51e7025702309dbb5e39 not found: ID does not exist" Nov 01 15:37:19.647324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.647232 5473 scope.go:122] "RemoveContainer" containerID="0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a" Nov 01 15:37:19.659106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.659090 5473 scope.go:122] "RemoveContainer" containerID="0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a" Nov 01 15:37:19.659385 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:37:19.659364 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a\": container with ID starting with 0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a not found: ID does not exist" containerID="0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a" Nov 01 15:37:19.659447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.659393 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a"} err="failed to get container status \"0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a\": rpc error: code = NotFound desc = could not find container \"0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a\": container with ID starting with 0ff1a1055b1beeaa9e07bc667fddc44fde496ce5718691610c044910b2cc527a not found: ID does not exist" Nov 01 15:37:19.659447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.659409 5473 scope.go:122] "RemoveContainer" containerID="3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a" Nov 01 15:37:19.669669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.669652 5473 scope.go:122] "RemoveContainer" containerID="3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a" Nov 01 15:37:19.669904 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:37:19.669886 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a\": container with ID starting with 3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a not found: ID does not exist" containerID="3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a" Nov 01 15:37:19.669949 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.669910 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a"} err="failed to get container status \"3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a\": rpc error: code = NotFound desc = could not find container \"3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a\": container with ID starting with 3439fbc3faa9ddb8378a812b3cb002b7443a91d65b779da935fa3f818ee2ad3a not found: ID does not exist" Nov 01 15:37:19.669949 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.669924 5473 scope.go:122] "RemoveContainer" containerID="90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1" Nov 01 15:37:19.679093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.679073 5473 scope.go:122] "RemoveContainer" containerID="90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1" Nov 01 15:37:19.679382 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:37:19.679363 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1\": container with ID starting with 90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1 not found: ID does not exist" containerID="90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1" Nov 01 15:37:19.679436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.679388 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1"} err="failed to get container status \"90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1\": rpc error: code = NotFound desc = could not find container \"90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1\": container with ID starting with 90b87825fe2f89d2b5241290d7d65dc5c02e128fdde00943f0b012fadc0d74b1 not found: ID does not exist" Nov 01 15:37:19.833230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.828239 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4"] Nov 01 15:37:19.833230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.833137 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lbfq4"] Nov 01 15:37:19.946950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.946910 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n"] Nov 01 15:37:19.950603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:19.950574 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-h9b2n"] Nov 01 15:37:20.058528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:20.058496 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw"] Nov 01 15:37:20.062101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:20.062043 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-stknw"] Nov 01 15:37:20.172850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:20.172820 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw"] Nov 01 15:37:20.176650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:20.176626 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-2zlpw"] Nov 01 15:37:21.083888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:21.083858 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" path="/var/lib/kubelet/pods/520cc1ca-a6c0-47ee-adf4-e288b0fb52d0/volumes" Nov 01 15:37:21.084397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:21.084381 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" path="/var/lib/kubelet/pods/c919f6f7-a12e-4fa4-a6d8-8b0adc060037/volumes" Nov 01 15:37:21.084843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:21.084830 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" path="/var/lib/kubelet/pods/ea4c58e2-dded-4c0a-82d7-23240410d460/volumes" Nov 01 15:37:21.085614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:21.085587 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" path="/var/lib/kubelet/pods/fb5c87e3-80a0-4c1b-827a-a7d78da38a99/volumes" Nov 01 15:37:23.087520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:23.087487 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:26.945848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:26.945812 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:28.522600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:28.522557 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:30.165687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:30.165646 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:30.711965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:30.711925 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:31.109850 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:37:31.109812 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:37:31.428345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:31.428209 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:32.868739 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:32.868709 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:33.035292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:33.035259 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:33.210627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:33.210537 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:37.610640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:37.610605 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:37.780643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:37.780605 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:38.861639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:38.861598 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:39.024344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:39.024302 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:41.492463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:41.492432 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:41.682415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:41.682378 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:42.383344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:42.383300 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:43.778007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:43.777973 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:44.667574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:44.667546 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:37:44.686936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:44.686897 5473 scope.go:122] "RemoveContainer" containerID="13d82e5934d39aebb6156d4b4d42d0626f249082776fdab411e01225225953eb" Nov 01 15:37:44.695489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:44.695469 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:37:46.857632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:46.857591 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:51.252238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:51.252182 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:37:55.755975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:37:55.755886 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:00.244408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:00.244366 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:01.109162 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:38:01.109115 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:38:04.595885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:04.595844 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:08.696025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:08.695959 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:09.023243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:09.023182 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:12.842592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:12.842560 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:14.474045 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:14.474008 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:18.198728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:18.198691 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:18.436436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:18.436403 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:19.172548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:19.172513 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:19.733192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:19.733160 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:20.359682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:20.359647 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:21.127136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:21.127099 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:24.155252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:24.155186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:27.027177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:27.027141 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:27.493843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:27.493761 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5"] Nov 01 15:38:27.600426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:27.600384 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5"] Nov 01 15:38:27.902992 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:27.902967 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:28.114975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:28.114943 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:28.722872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:28.722824 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:31.110680 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:38:31.110645 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:38:32.494894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.494856 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" podUID="c75dd3c8-9357-48da-9daf-72f409718749" containerName="payload-processing" containerID="cri-o://1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e" gracePeriod=25 Nov 01 15:38:32.601633 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.601587 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" podUID="e1bbdd6a-7154-4bc5-915a-874519683280" containerName="payload-pre-processing" containerID="cri-o://f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1" gracePeriod=25 Nov 01 15:38:32.782343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.781457 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:38:32.881862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.881832 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume\") pod \"c75dd3c8-9357-48da-9daf-72f409718749\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " Nov 01 15:38:32.882012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.881952 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m\" (UniqueName: \"kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m\") pod \"c75dd3c8-9357-48da-9daf-72f409718749\" (UID: \"c75dd3c8-9357-48da-9daf-72f409718749\") " Nov 01 15:38:32.882188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.882163 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume" pod "c75dd3c8-9357-48da-9daf-72f409718749" (UID: "c75dd3c8-9357-48da-9daf-72f409718749"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:38:32.882561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.882540 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c75dd3c8-9357-48da-9daf-72f409718749-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:38:32.884160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.884135 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m" pod "c75dd3c8-9357-48da-9daf-72f409718749" (UID: "c75dd3c8-9357-48da-9daf-72f409718749"). InnerVolumeSpecName "kube-api-access-dkm8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:38:32.893478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.893458 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:38:32.983317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.983285 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume\") pod \"e1bbdd6a-7154-4bc5-915a-874519683280\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " Nov 01 15:38:32.983501 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.983402 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn\" (UniqueName: \"kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn\") pod \"e1bbdd6a-7154-4bc5-915a-874519683280\" (UID: \"e1bbdd6a-7154-4bc5-915a-874519683280\") " Nov 01 15:38:32.983657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.983635 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume" pod "e1bbdd6a-7154-4bc5-915a-874519683280" (UID: "e1bbdd6a-7154-4bc5-915a-874519683280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:38:32.983909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.983891 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1bbdd6a-7154-4bc5-915a-874519683280-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:38:32.983956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.983914 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkm8m\" (UniqueName: \"kubernetes.io/projected/c75dd3c8-9357-48da-9daf-72f409718749-kube-api-access-dkm8m\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:38:32.985565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.985545 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn" pod "e1bbdd6a-7154-4bc5-915a-874519683280" (UID: "e1bbdd6a-7154-4bc5-915a-874519683280"). InnerVolumeSpecName "kube-api-access-r7vrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:38:32.998485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:32.998430 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:33.084921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.084885 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vrn\" (UniqueName: \"kubernetes.io/projected/e1bbdd6a-7154-4bc5-915a-874519683280-kube-api-access-r7vrn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:38:33.249657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.249567 5473 generic.go:356] "Generic (PLEG): container finished" podID="c75dd3c8-9357-48da-9daf-72f409718749" containerID="1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e" exitCode=0 Nov 01 15:38:33.249657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.249621 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" Nov 01 15:38:33.249881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.249652 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" event={"ID":"c75dd3c8-9357-48da-9daf-72f409718749","Type":"ContainerDied","Data":"1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e"} Nov 01 15:38:33.249881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.249733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5" event={"ID":"c75dd3c8-9357-48da-9daf-72f409718749","Type":"ContainerDied","Data":"62a1bca3a9d534a70450c5ddf6c21d1e834b5b4ba6c24e5f7c1130ae8a202265"} Nov 01 15:38:33.249881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.249759 5473 scope.go:122] "RemoveContainer" containerID="1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e" Nov 01 15:38:33.251322 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.251297 5473 generic.go:356] "Generic (PLEG): container finished" podID="e1bbdd6a-7154-4bc5-915a-874519683280" containerID="f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1" exitCode=0 Nov 01 15:38:33.251452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.251347 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" Nov 01 15:38:33.251452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.251355 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" event={"ID":"e1bbdd6a-7154-4bc5-915a-874519683280","Type":"ContainerDied","Data":"f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1"} Nov 01 15:38:33.251452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.251397 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5" event={"ID":"e1bbdd6a-7154-4bc5-915a-874519683280","Type":"ContainerDied","Data":"f84afddc64ce4bc14f53234df3084864bee9c19fee3988590b986feec043c11f"} Nov 01 15:38:33.260665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.260644 5473 scope.go:122] "RemoveContainer" containerID="1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e" Nov 01 15:38:33.260934 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:38:33.260913 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e\": container with ID starting with 1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e not found: ID does not exist" containerID="1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e" Nov 01 15:38:33.261009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.260944 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e"} err="failed to get container status \"1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e\": rpc error: code = NotFound desc = could not find container \"1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e\": container with ID starting with 1a811a575471e20e651d3a41358357f87ae606bc182e708486adaabc2a6e0f5e not found: ID does not exist" Nov 01 15:38:33.261009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.260967 5473 scope.go:122] "RemoveContainer" containerID="f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1" Nov 01 15:38:33.270436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.270418 5473 scope.go:122] "RemoveContainer" containerID="f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1" Nov 01 15:38:33.270688 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:38:33.270666 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1\": container with ID starting with f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1 not found: ID does not exist" containerID="f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1" Nov 01 15:38:33.270754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.270700 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1"} err="failed to get container status \"f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1\": rpc error: code = NotFound desc = could not find container \"f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1\": container with ID starting with f0cb58fafb40660e822a41e228af88a7e16f3d002db439fb17b45efadef497f1 not found: ID does not exist" Nov 01 15:38:33.307684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.307659 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:33.418320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.418281 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5"] Nov 01 15:38:33.420892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.420869 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-974865-68f66d5c4c-wkxg5"] Nov 01 15:38:33.530973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.530944 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5"] Nov 01 15:38:33.537776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:33.537751 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-974865-5b76c87d77-lm4d5"] Nov 01 15:38:35.082576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:35.082543 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75dd3c8-9357-48da-9daf-72f409718749" path="/var/lib/kubelet/pods/c75dd3c8-9357-48da-9daf-72f409718749/volumes" Nov 01 15:38:35.082970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:35.082954 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1bbdd6a-7154-4bc5-915a-874519683280" path="/var/lib/kubelet/pods/e1bbdd6a-7154-4bc5-915a-874519683280/volumes" Nov 01 15:38:37.708209 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:37.708173 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:42.194465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:42.194431 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:46.593899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:46.593861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:50.450265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:50.450138 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:52.147433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:52.147395 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:56.519827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:56.519796 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:58.933390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.933358 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp"] Nov 01 15:38:58.933967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.933953 5473 cpu_manager.go:414] "containerMap: removing container" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" Nov 01 15:38:58.933967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.933967 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.933983 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c75dd3c8-9357-48da-9daf-72f409718749" containerName="payload-processing" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.933988 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75dd3c8-9357-48da-9daf-72f409718749" containerName="payload-processing" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934008 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934014 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934022 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934027 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934038 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" Nov 01 15:38:58.934063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934043 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934069 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e1bbdd6a-7154-4bc5-915a-874519683280" containerName="payload-pre-processing" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934074 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1bbdd6a-7154-4bc5-915a-874519683280" containerName="payload-pre-processing" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934162 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1bbdd6a-7154-4bc5-915a-874519683280" containerName="payload-pre-processing" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934172 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="520cc1ca-a6c0-47ee-adf4-e288b0fb52d0" containerName="istio-proxy" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934184 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75dd3c8-9357-48da-9daf-72f409718749" containerName="payload-processing" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934196 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5c87e3-80a0-4c1b-827a-a7d78da38a99" containerName="istio-proxy" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934203 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4c58e2-dded-4c0a-82d7-23240410d460" containerName="istio-proxy" Nov 01 15:38:58.934352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.934210 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c919f6f7-a12e-4fa4-a6d8-8b0adc060037" containerName="istio-proxy" Nov 01 15:38:58.937746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.937727 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:58.995479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:58.995454 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp"] Nov 01 15:38:59.040713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.040679 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:38:59.040865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.040775 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:38:59.111017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.110989 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.111017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.111023 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z2lj\" (UniqueName: \"kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.111235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.111202 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.212903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.212829 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.213071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.212924 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.213071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.212945 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z2lj\" (UniqueName: \"kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.213758 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.213731 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.215266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.215245 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.269933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.269900 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z2lj\" (UniqueName: \"kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj\") pod \"maas-api-revoke-sub-029de10c-db9qp\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.554003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.553968 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:38:59.879637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.879559 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:38:59.973310 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:38:59.958440 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff71f8d_a340_4686_b832_236fc4b2728f.slice/crio-e0f560de4ab77168bd1bc67405f53ee9d7aafe9857ac71a63bbeac7f5de86a76 WatchSource:0}: Error finding container e0f560de4ab77168bd1bc67405f53ee9d7aafe9857ac71a63bbeac7f5de86a76: Status 404 returned error can't find the container with id e0f560de4ab77168bd1bc67405f53ee9d7aafe9857ac71a63bbeac7f5de86a76 Nov 01 15:38:59.999024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:38:59.997903 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp"] Nov 01 15:39:00.044536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:00.044435 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:00.454116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:00.454078 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" event={"ID":"eff71f8d-a340-4686-b832-236fc4b2728f","Type":"ContainerStarted","Data":"267444ac40320d77196baa6a39cec43d7b663bdd45cc5d07ade5df85a2f1a003"} Nov 01 15:39:00.454311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:00.454135 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" event={"ID":"eff71f8d-a340-4686-b832-236fc4b2728f","Type":"ContainerStarted","Data":"e0f560de4ab77168bd1bc67405f53ee9d7aafe9857ac71a63bbeac7f5de86a76"} Nov 01 15:39:00.571688 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:00.571631 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" podStartSLOduration=2.5716178530000002 podStartE2EDuration="2.571617853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:39:00.56861095 +0000 UTC m=+4879.626762804" watchObservedRunningTime="2026-11-01 15:39:00.571617853 +0000 UTC m=+4879.629769704" Nov 01 15:39:01.091396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:01.091364 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:01.118781 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:39:01.118729 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:39:01.463436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:01.463400 5473 generic.go:356] "Generic (PLEG): container finished" podID="eff71f8d-a340-4686-b832-236fc4b2728f" containerID="267444ac40320d77196baa6a39cec43d7b663bdd45cc5d07ade5df85a2f1a003" exitCode=0 Nov 01 15:39:01.463630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:01.463478 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" event={"ID":"eff71f8d-a340-4686-b832-236fc4b2728f","Type":"ContainerDied","Data":"267444ac40320d77196baa6a39cec43d7b663bdd45cc5d07ade5df85a2f1a003"} Nov 01 15:39:01.543503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:01.543470 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:01.964742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:01.964707 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t"] Nov 01 15:39:01.964975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:01.964953 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" containerID="cri-o://b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320" gracePeriod=30 Nov 01 15:39:02.020367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.020316 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:02.623079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.623055 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:39:02.680034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.679987 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca\") pod \"eff71f8d-a340-4686-b832-236fc4b2728f\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " Nov 01 15:39:02.680274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.680062 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj\" (UniqueName: \"kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj\") pod \"eff71f8d-a340-4686-b832-236fc4b2728f\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " Nov 01 15:39:02.680383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.680281 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca\") pod \"eff71f8d-a340-4686-b832-236fc4b2728f\" (UID: \"eff71f8d-a340-4686-b832-236fc4b2728f\") " Nov 01 15:39:02.680871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.680832 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca" pod "eff71f8d-a340-4686-b832-236fc4b2728f" (UID: "eff71f8d-a340-4686-b832-236fc4b2728f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:02.682659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.682629 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca" pod "eff71f8d-a340-4686-b832-236fc4b2728f" (UID: "eff71f8d-a340-4686-b832-236fc4b2728f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:02.682659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.682644 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj" pod "eff71f8d-a340-4686-b832-236fc4b2728f" (UID: "eff71f8d-a340-4686-b832-236fc4b2728f"). InnerVolumeSpecName "kube-api-access-2z2lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:02.721614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.721584 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:02.782066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.782038 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:02.782066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.782063 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z2lj\" (UniqueName: \"kubernetes.io/projected/eff71f8d-a340-4686-b832-236fc4b2728f-kube-api-access-2z2lj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:02.782309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:02.782076 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff71f8d-a340-4686-b832-236fc4b2728f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:03.480708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:03.480626 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" Nov 01 15:39:03.480708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:03.480654 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp" event={"ID":"eff71f8d-a340-4686-b832-236fc4b2728f","Type":"ContainerDied","Data":"e0f560de4ab77168bd1bc67405f53ee9d7aafe9857ac71a63bbeac7f5de86a76"} Nov 01 15:39:03.480708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:03.480704 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0f560de4ab77168bd1bc67405f53ee9d7aafe9857ac71a63bbeac7f5de86a76" Nov 01 15:39:05.925749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:05.925710 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:07.232682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.232621 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 15:39:07.341524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341494 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.341681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341562 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.341681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341582 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.341681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341628 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.341851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341679 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.341851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341719 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.341851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341769 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.342024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341838 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:07.342024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.341912 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:07.342149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342075 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:07.342149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342096 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:07.342149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342130 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:07.342359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342165 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.342359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342196 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl\") pod \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\" (UID: \"80934e4d-60a9-410d-9e9d-8ac3320fa74d\") " Nov 01 15:39:07.342718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342699 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.342806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342719 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.342806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342732 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.342806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342744 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.342806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.342757 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.343975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.343941 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:07.344198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.344177 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:39:07.344423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.344404 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:07.344556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.344538 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl" pod "80934e4d-60a9-410d-9e9d-8ac3320fa74d" (UID: "80934e4d-60a9-410d-9e9d-8ac3320fa74d"). InnerVolumeSpecName "kube-api-access-psljl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:07.443665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.443629 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.443665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.443661 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.443665 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.443673 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psljl\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-kube-api-access-psljl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.443911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.443684 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80934e4d-60a9-410d-9e9d-8ac3320fa74d-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:07.514036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.514007 5473 generic.go:356] "Generic (PLEG): container finished" podID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerID="b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320" exitCode=0 Nov 01 15:39:07.514207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.514055 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" Nov 01 15:39:07.514207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.514093 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" event={"ID":"80934e4d-60a9-410d-9e9d-8ac3320fa74d","Type":"ContainerDied","Data":"b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320"} Nov 01 15:39:07.514207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.514154 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t" event={"ID":"80934e4d-60a9-410d-9e9d-8ac3320fa74d","Type":"ContainerDied","Data":"c9ca166edc34362c09c6eef17003a6a622565bcac133d018d7e74b2a391439bc"} Nov 01 15:39:07.514207 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.514176 5473 scope.go:122] "RemoveContainer" containerID="b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320" Nov 01 15:39:07.525792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.525768 5473 scope.go:122] "RemoveContainer" containerID="bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77" Nov 01 15:39:07.535624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.535606 5473 scope.go:122] "RemoveContainer" containerID="b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320" Nov 01 15:39:07.535895 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:39:07.535873 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320\": container with ID starting with b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320 not found: ID does not exist" containerID="b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320" Nov 01 15:39:07.535958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.535902 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320"} err="failed to get container status \"b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320\": rpc error: code = NotFound desc = could not find container \"b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320\": container with ID starting with b9d8b492616dbc3dc6079975abebc880649606f754d4207e3999003ed60f8320 not found: ID does not exist" Nov 01 15:39:07.535958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.535922 5473 scope.go:122] "RemoveContainer" containerID="bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77" Nov 01 15:39:07.536185 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:39:07.536165 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77\": container with ID starting with bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77 not found: ID does not exist" containerID="bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77" Nov 01 15:39:07.536269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.536190 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77"} err="failed to get container status \"bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77\": rpc error: code = NotFound desc = could not find container \"bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77\": container with ID starting with bed336b1795640eba96c68d0e995b8b3ebdbdedb9e9475ff99996369bd84aa77 not found: ID does not exist" Nov 01 15:39:07.681266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.681232 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t"] Nov 01 15:39:07.684947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:07.684920 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d06633-openshift-default-6d5ccbff8c-8th7t"] Nov 01 15:39:08.922009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:08.921977 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:09.082896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.082866 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" path="/var/lib/kubelet/pods/80934e4d-60a9-410d-9e9d-8ac3320fa74d/volumes" Nov 01 15:39:09.087843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.087817 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:09.602107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.602069 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:09.733264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733210 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz"] Nov 01 15:39:09.733887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733872 5473 cpu_manager.go:414] "containerMap: removing container" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" Nov 01 15:39:09.733946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733888 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" Nov 01 15:39:09.733946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733912 5473 cpu_manager.go:414] "containerMap: removing container" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" Nov 01 15:39:09.733946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733918 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" Nov 01 15:39:09.733946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733931 5473 cpu_manager.go:414] "containerMap: removing container" podUID="eff71f8d-a340-4686-b832-236fc4b2728f" containerName="revoke-keys" Nov 01 15:39:09.733946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.733936 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff71f8d-a340-4686-b832-236fc4b2728f" containerName="revoke-keys" Nov 01 15:39:09.734160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.734133 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" Nov 01 15:39:09.734160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.734153 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff71f8d-a340-4686-b832-236fc4b2728f" containerName="revoke-keys" Nov 01 15:39:09.741226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.741180 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.761348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.761315 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:09.771848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.771823 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.772048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.772028 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.772150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.772134 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrscp\" (UniqueName: \"kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.793522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.793488 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz"] Nov 01 15:39:09.839448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.839409 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:39:09.839635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.839498 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:39:09.873062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.872977 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.873262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.873101 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.873262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.873149 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrscp\" (UniqueName: \"kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.873738 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.873712 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.875489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.875467 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:09.930320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:09.930292 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrscp\" (UniqueName: \"kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp\") pod \"maas-api-revoke-sub-543f285d-k54lz\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:10.060377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:10.060343 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:10.442713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:10.442687 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz"] Nov 01 15:39:10.443878 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:10.443848 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bff4d90_0ffa_4a57_bc9c_38d5e8ad53ff.slice/crio-5927637201f019a297800da31439296d8f95615ac2aecff5aeb7567af1480f26 WatchSource:0}: Error finding container 5927637201f019a297800da31439296d8f95615ac2aecff5aeb7567af1480f26: Status 404 returned error can't find the container with id 5927637201f019a297800da31439296d8f95615ac2aecff5aeb7567af1480f26 Nov 01 15:39:10.468458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:10.468432 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:10.539684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:10.539645 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" event={"ID":"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff","Type":"ContainerStarted","Data":"5927637201f019a297800da31439296d8f95615ac2aecff5aeb7567af1480f26"} Nov 01 15:39:11.548956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:11.548923 5473 generic.go:356] "Generic (PLEG): container finished" podID="7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" containerID="5b1874c2d6591a76441d9f6aa3eeffeb63ba921efd0a43cb1c67e14b5fe2d816" exitCode=0 Nov 01 15:39:11.549383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:11.549012 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" event={"ID":"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff","Type":"ContainerDied","Data":"5b1874c2d6591a76441d9f6aa3eeffeb63ba921efd0a43cb1c67e14b5fe2d816"} Nov 01 15:39:12.703691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.703666 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:12.812350 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.812273 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca\") pod \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " Nov 01 15:39:12.812529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.812405 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca\") pod \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " Nov 01 15:39:12.812529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.812474 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp\" (UniqueName: \"kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp\") pod \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\" (UID: \"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff\") " Nov 01 15:39:12.812700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.812676 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca" pod "7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" (UID: "7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:12.812815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.812801 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:12.814393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.814370 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca" pod "7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" (UID: "7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:12.814482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.814447 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp" pod "7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" (UID: "7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff"). InnerVolumeSpecName "kube-api-access-jrscp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:12.914691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.914652 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:12.914691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:12.914687 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrscp\" (UniqueName: \"kubernetes.io/projected/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff-kube-api-access-jrscp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:13.563579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:13.563552 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" Nov 01 15:39:13.563795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:13.563589 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz" event={"ID":"7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff","Type":"ContainerDied","Data":"5927637201f019a297800da31439296d8f95615ac2aecff5aeb7567af1480f26"} Nov 01 15:39:13.563795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:13.563666 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5927637201f019a297800da31439296d8f95615ac2aecff5aeb7567af1480f26" Nov 01 15:39:14.687141 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:14.687101 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:15.015390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:15.015351 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:19.403413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:19.403384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:21.292555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.292519 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw"] Nov 01 15:39:21.293232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.293198 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" containerName="revoke-keys" Nov 01 15:39:21.293232 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.293230 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" containerName="revoke-keys" Nov 01 15:39:21.293369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.293358 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="80934e4d-60a9-410d-9e9d-8ac3320fa74d" containerName="istio-proxy" Nov 01 15:39:21.293408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.293371 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" containerName="revoke-keys" Nov 01 15:39:21.296879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.296861 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.352277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.352246 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw"] Nov 01 15:39:21.397520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.397488 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:39:21.397712 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.397557 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:39:21.427376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.427344 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.427528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.427394 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.427528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.427454 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmk5\" (UniqueName: \"kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.528685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.528650 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.528889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.528704 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.528889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.528730 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmk5\" (UniqueName: \"kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.529339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.529316 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.531279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.531258 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.585664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.585583 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmk5\" (UniqueName: \"kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5\") pod \"maas-api-revoke-sub-e2f6319b-ml6xw\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.611669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.611636 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:21.998415 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:21.998372 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f861b9a_c205_402e_ad42_452194b30128.slice/crio-ce648a2217c543155a13939213081c02f1e792e1163580c68d66cdde509ec97c WatchSource:0}: Error finding container ce648a2217c543155a13939213081c02f1e792e1163580c68d66cdde509ec97c: Status 404 returned error can't find the container with id ce648a2217c543155a13939213081c02f1e792e1163580c68d66cdde509ec97c Nov 01 15:39:21.998565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:21.998531 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw"] Nov 01 15:39:22.634525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:22.634442 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" event={"ID":"6f861b9a-c205-402e-ad42-452194b30128","Type":"ContainerStarted","Data":"cb86d055379ded9b28a7564ae9495fb81523cedc47ff9575f0b6db254845988f"} Nov 01 15:39:22.634525 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:22.634494 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" event={"ID":"6f861b9a-c205-402e-ad42-452194b30128","Type":"ContainerStarted","Data":"ce648a2217c543155a13939213081c02f1e792e1163580c68d66cdde509ec97c"} Nov 01 15:39:22.751085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:22.751034 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" podStartSLOduration=1.751016009 podStartE2EDuration="1.751016009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:39:22.749096082 +0000 UTC m=+4901.807247944" watchObservedRunningTime="2026-11-01 15:39:22.751016009 +0000 UTC m=+4901.809167858" Nov 01 15:39:23.645057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:23.645014 5473 generic.go:356] "Generic (PLEG): container finished" podID="6f861b9a-c205-402e-ad42-452194b30128" containerID="cb86d055379ded9b28a7564ae9495fb81523cedc47ff9575f0b6db254845988f" exitCode=0 Nov 01 15:39:23.645528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:23.645071 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" event={"ID":"6f861b9a-c205-402e-ad42-452194b30128","Type":"ContainerDied","Data":"cb86d055379ded9b28a7564ae9495fb81523cedc47ff9575f0b6db254845988f"} Nov 01 15:39:23.841694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:23.841618 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:24.808367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.808344 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:24.973722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.973642 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca\") pod \"6f861b9a-c205-402e-ad42-452194b30128\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " Nov 01 15:39:24.973722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.973717 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca\") pod \"6f861b9a-c205-402e-ad42-452194b30128\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " Nov 01 15:39:24.973925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.973737 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5\" (UniqueName: \"kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5\") pod \"6f861b9a-c205-402e-ad42-452194b30128\" (UID: \"6f861b9a-c205-402e-ad42-452194b30128\") " Nov 01 15:39:24.974065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.974038 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca" pod "6f861b9a-c205-402e-ad42-452194b30128" (UID: "6f861b9a-c205-402e-ad42-452194b30128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:24.974189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.974171 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f861b9a-c205-402e-ad42-452194b30128-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:24.975831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.975813 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca" pod "6f861b9a-c205-402e-ad42-452194b30128" (UID: "6f861b9a-c205-402e-ad42-452194b30128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:24.975946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:24.975931 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5" pod "6f861b9a-c205-402e-ad42-452194b30128" (UID: "6f861b9a-c205-402e-ad42-452194b30128"). InnerVolumeSpecName "kube-api-access-jxmk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:25.075324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:25.075289 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f861b9a-c205-402e-ad42-452194b30128-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:25.075324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:25.075320 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmk5\" (UniqueName: \"kubernetes.io/projected/6f861b9a-c205-402e-ad42-452194b30128-kube-api-access-jxmk5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:25.659900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:25.659867 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" Nov 01 15:39:25.659900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:25.659892 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw" event={"ID":"6f861b9a-c205-402e-ad42-452194b30128","Type":"ContainerDied","Data":"ce648a2217c543155a13939213081c02f1e792e1163580c68d66cdde509ec97c"} Nov 01 15:39:25.660137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:25.659971 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce648a2217c543155a13939213081c02f1e792e1163580c68d66cdde509ec97c" Nov 01 15:39:27.078767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:27.078734 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:27.257693 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:27.257656 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:28.279457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:28.279426 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:30.017290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.017257 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm"] Nov 01 15:39:30.017902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.017888 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6f861b9a-c205-402e-ad42-452194b30128" containerName="revoke-keys" Nov 01 15:39:30.017902 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.017903 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f861b9a-c205-402e-ad42-452194b30128" containerName="revoke-keys" Nov 01 15:39:30.018032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.018023 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f861b9a-c205-402e-ad42-452194b30128" containerName="revoke-keys" Nov 01 15:39:30.021541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.021518 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.039802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.039773 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.039948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.039929 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.040066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.040022 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84c2\" (UniqueName: \"kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.078018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.077983 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm"] Nov 01 15:39:30.123006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.122974 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:39:30.123191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.123091 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:39:30.141308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.141273 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.141469 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.141361 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84c2\" (UniqueName: \"kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.141732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.141707 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.142397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.142375 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.143870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.143843 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.197832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.197800 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84c2\" (UniqueName: \"kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2\") pod \"maas-api-revoke-sub-8e6f39f2-ddfjm\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.335610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.335527 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:30.716950 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:30.716919 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2506ab_b32e_4b41_81b3_f3e54dc10bf2.slice/crio-1170b162a74fb9ad02e07d63cf30ebe94ca09fc3a5255d84a080b26af4e869b6 WatchSource:0}: Error finding container 1170b162a74fb9ad02e07d63cf30ebe94ca09fc3a5255d84a080b26af4e869b6: Status 404 returned error can't find the container with id 1170b162a74fb9ad02e07d63cf30ebe94ca09fc3a5255d84a080b26af4e869b6 Nov 01 15:39:30.718033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:30.718000 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm"] Nov 01 15:39:31.117614 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:39:31.117573 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:39:31.705492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:31.705456 5473 generic.go:356] "Generic (PLEG): container finished" podID="2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" containerID="10f3c48ced753d95412ca5c6581eef9d980a909c470219d9358b489a6c6922b7" exitCode=0 Nov 01 15:39:31.705682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:31.705541 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" event={"ID":"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2","Type":"ContainerDied","Data":"10f3c48ced753d95412ca5c6581eef9d980a909c470219d9358b489a6c6922b7"} Nov 01 15:39:31.705682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:31.705604 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" event={"ID":"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2","Type":"ContainerStarted","Data":"1170b162a74fb9ad02e07d63cf30ebe94ca09fc3a5255d84a080b26af4e869b6"} Nov 01 15:39:32.179563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.179523 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn"] Nov 01 15:39:32.184150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.184128 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.240615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.240574 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn"] Nov 01 15:39:32.266728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.266688 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.266938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.266851 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.267013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.266975 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnnc\" (UniqueName: \"kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.368740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.368702 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.368925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.368811 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnnc\" (UniqueName: \"kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.368925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.368858 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.369386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.369363 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.371314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.371289 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.425980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.425945 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnnc\" (UniqueName: \"kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc\") pod \"maas-api-revoke-sub-e7b795f7-99hvn\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.497760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.497671 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:32.760846 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.760759 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:32.874992 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:32.874963 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c806575_f331_440c_97f7_ec39ef4d975d.slice/crio-1d1fd2493244d252b27e71743948d49806b023968475149ae1a4da53c5868024 WatchSource:0}: Error finding container 1d1fd2493244d252b27e71743948d49806b023968475149ae1a4da53c5868024: Status 404 returned error can't find the container with id 1d1fd2493244d252b27e71743948d49806b023968475149ae1a4da53c5868024 Nov 01 15:39:32.875496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.875466 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn"] Nov 01 15:39:32.889307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.889277 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:32.976300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.976211 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca\") pod \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " Nov 01 15:39:32.976502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.976375 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2\" (UniqueName: \"kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2\") pod \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " Nov 01 15:39:32.976502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.976449 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca\") pod \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\" (UID: \"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2\") " Nov 01 15:39:32.976827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.976797 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca" pod "2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" (UID: "2e2506ab-b32e-4b41-81b3-f3e54dc10bf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:32.978463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.978442 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2" pod "2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" (UID: "2e2506ab-b32e-4b41-81b3-f3e54dc10bf2"). InnerVolumeSpecName "kube-api-access-m84c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:32.978463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:32.978450 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca" pod "2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" (UID: "2e2506ab-b32e-4b41-81b3-f3e54dc10bf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:33.077865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.077827 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84c2\" (UniqueName: \"kubernetes.io/projected/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-kube-api-access-m84c2\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:33.077865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.077855 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:33.077865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.077865 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:33.722413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.722322 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" Nov 01 15:39:33.722413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.722335 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm" event={"ID":"2e2506ab-b32e-4b41-81b3-f3e54dc10bf2","Type":"ContainerDied","Data":"1170b162a74fb9ad02e07d63cf30ebe94ca09fc3a5255d84a080b26af4e869b6"} Nov 01 15:39:33.722413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.722387 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1170b162a74fb9ad02e07d63cf30ebe94ca09fc3a5255d84a080b26af4e869b6" Nov 01 15:39:33.724158 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.724123 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" event={"ID":"9c806575-f331-440c-97f7-ec39ef4d975d","Type":"ContainerStarted","Data":"522c143ec4ba03f47772ec6e31a029c1524baa7d7e4e8681bb026f247a6290d8"} Nov 01 15:39:33.724303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.724176 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" event={"ID":"9c806575-f331-440c-97f7-ec39ef4d975d","Type":"ContainerStarted","Data":"1d1fd2493244d252b27e71743948d49806b023968475149ae1a4da53c5868024"} Nov 01 15:39:33.837822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:33.837754 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" podStartSLOduration=1.837738078 podStartE2EDuration="1.837738078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:39:33.837091069 +0000 UTC m=+4912.895242921" watchObservedRunningTime="2026-11-01 15:39:33.837738078 +0000 UTC m=+4912.895889928" Nov 01 15:39:34.733662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:34.733620 5473 generic.go:356] "Generic (PLEG): container finished" podID="9c806575-f331-440c-97f7-ec39ef4d975d" containerID="522c143ec4ba03f47772ec6e31a029c1524baa7d7e4e8681bb026f247a6290d8" exitCode=0 Nov 01 15:39:34.734159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:34.733812 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" event={"ID":"9c806575-f331-440c-97f7-ec39ef4d975d","Type":"ContainerDied","Data":"522c143ec4ba03f47772ec6e31a029c1524baa7d7e4e8681bb026f247a6290d8"} Nov 01 15:39:35.885479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:35.885450 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:36.013139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.013111 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca\") pod \"9c806575-f331-440c-97f7-ec39ef4d975d\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " Nov 01 15:39:36.013464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.013441 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc\" (UniqueName: \"kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc\") pod \"9c806575-f331-440c-97f7-ec39ef4d975d\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " Nov 01 15:39:36.013581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.013529 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca\") pod \"9c806575-f331-440c-97f7-ec39ef4d975d\" (UID: \"9c806575-f331-440c-97f7-ec39ef4d975d\") " Nov 01 15:39:36.013861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.013833 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca" pod "9c806575-f331-440c-97f7-ec39ef4d975d" (UID: "9c806575-f331-440c-97f7-ec39ef4d975d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:36.014099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.014078 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:36.015400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.015375 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca" pod "9c806575-f331-440c-97f7-ec39ef4d975d" (UID: "9c806575-f331-440c-97f7-ec39ef4d975d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:36.015583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.015567 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc" pod "9c806575-f331-440c-97f7-ec39ef4d975d" (UID: "9c806575-f331-440c-97f7-ec39ef4d975d"). InnerVolumeSpecName "kube-api-access-jtnnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:36.115792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.115755 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtnnc\" (UniqueName: \"kubernetes.io/projected/9c806575-f331-440c-97f7-ec39ef4d975d-kube-api-access-jtnnc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:36.115792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.115787 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c806575-f331-440c-97f7-ec39ef4d975d-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:36.749706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.749679 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" Nov 01 15:39:36.749706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.749686 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn" event={"ID":"9c806575-f331-440c-97f7-ec39ef4d975d","Type":"ContainerDied","Data":"1d1fd2493244d252b27e71743948d49806b023968475149ae1a4da53c5868024"} Nov 01 15:39:36.749935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:36.749739 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d1fd2493244d252b27e71743948d49806b023968475149ae1a4da53c5868024" Nov 01 15:39:37.143136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:37.143103 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:40.693616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.693579 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng"] Nov 01 15:39:40.694305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.694275 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" containerName="revoke-keys" Nov 01 15:39:40.694305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.694305 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" containerName="revoke-keys" Nov 01 15:39:40.694426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.694330 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9c806575-f331-440c-97f7-ec39ef4d975d" containerName="revoke-keys" Nov 01 15:39:40.694426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.694336 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c806575-f331-440c-97f7-ec39ef4d975d" containerName="revoke-keys" Nov 01 15:39:40.694493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.694459 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" containerName="revoke-keys" Nov 01 15:39:40.694493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.694471 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c806575-f331-440c-97f7-ec39ef4d975d" containerName="revoke-keys" Nov 01 15:39:40.698210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.698192 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.754172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.754129 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng"] Nov 01 15:39:40.775898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.775867 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkwz\" (UniqueName: \"kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.776133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.775926 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.776133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.776076 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.800046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.800005 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:39:40.800233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.800070 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:39:40.877734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.877696 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkwz\" (UniqueName: \"kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.877922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.877746 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.877922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.877825 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.878517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.878490 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.880315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.880298 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:40.934791 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:40.934761 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkwz\" (UniqueName: \"kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz\") pod \"maas-api-revoke-sub-d2031217-46lng\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:41.012380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:41.012356 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:41.386354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:41.386325 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng"] Nov 01 15:39:41.386629 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:41.386607 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c085104_f3de_4d3f_abc3_3be9c57585d3.slice/crio-51530d20c8e5e29975f5e39cef9ebe154b73a31b25f749086836b8b4acbee038 WatchSource:0}: Error finding container 51530d20c8e5e29975f5e39cef9ebe154b73a31b25f749086836b8b4acbee038: Status 404 returned error can't find the container with id 51530d20c8e5e29975f5e39cef9ebe154b73a31b25f749086836b8b4acbee038 Nov 01 15:39:41.787559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:41.787521 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" event={"ID":"6c085104-f3de-4d3f-abc3-3be9c57585d3","Type":"ContainerStarted","Data":"3014fcd59ff7f07942cc3621299731e57936522109ba93937092118fe3960f22"} Nov 01 15:39:41.787978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:41.787581 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" event={"ID":"6c085104-f3de-4d3f-abc3-3be9c57585d3","Type":"ContainerStarted","Data":"51530d20c8e5e29975f5e39cef9ebe154b73a31b25f749086836b8b4acbee038"} Nov 01 15:39:41.903259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:41.903154 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" podStartSLOduration=1.9031406469999999 podStartE2EDuration="1.903140647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:39:41.900406324 +0000 UTC m=+4920.958558174" watchObservedRunningTime="2026-11-01 15:39:41.903140647 +0000 UTC m=+4920.961292546" Nov 01 15:39:42.798085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:42.798047 5473 generic.go:356] "Generic (PLEG): container finished" podID="6c085104-f3de-4d3f-abc3-3be9c57585d3" containerID="3014fcd59ff7f07942cc3621299731e57936522109ba93937092118fe3960f22" exitCode=0 Nov 01 15:39:42.798496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:42.798125 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" event={"ID":"6c085104-f3de-4d3f-abc3-3be9c57585d3","Type":"ContainerDied","Data":"3014fcd59ff7f07942cc3621299731e57936522109ba93937092118fe3960f22"} Nov 01 15:39:43.491472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:43.491437 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:43.962621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:43.962597 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:44.123051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.122973 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca\") pod \"6c085104-f3de-4d3f-abc3-3be9c57585d3\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " Nov 01 15:39:44.123200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.123178 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca\") pod \"6c085104-f3de-4d3f-abc3-3be9c57585d3\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " Nov 01 15:39:44.123261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.123243 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz\" (UniqueName: \"kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz\") pod \"6c085104-f3de-4d3f-abc3-3be9c57585d3\" (UID: \"6c085104-f3de-4d3f-abc3-3be9c57585d3\") " Nov 01 15:39:44.123537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.123502 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca" pod "6c085104-f3de-4d3f-abc3-3be9c57585d3" (UID: "6c085104-f3de-4d3f-abc3-3be9c57585d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:44.123663 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.123606 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:44.125160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.125138 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca" pod "6c085104-f3de-4d3f-abc3-3be9c57585d3" (UID: "6c085104-f3de-4d3f-abc3-3be9c57585d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:44.125500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.125480 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz" pod "6c085104-f3de-4d3f-abc3-3be9c57585d3" (UID: "6c085104-f3de-4d3f-abc3-3be9c57585d3"). InnerVolumeSpecName "kube-api-access-hhkwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:44.224733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.224697 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhkwz\" (UniqueName: \"kubernetes.io/projected/6c085104-f3de-4d3f-abc3-3be9c57585d3-kube-api-access-hhkwz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:44.224733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.224732 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c085104-f3de-4d3f-abc3-3be9c57585d3-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:44.232607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.232578 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:44.814110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.814078 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" Nov 01 15:39:44.814306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.814078 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng" event={"ID":"6c085104-f3de-4d3f-abc3-3be9c57585d3","Type":"ContainerDied","Data":"51530d20c8e5e29975f5e39cef9ebe154b73a31b25f749086836b8b4acbee038"} Nov 01 15:39:44.814306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:44.814198 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51530d20c8e5e29975f5e39cef9ebe154b73a31b25f749086836b8b4acbee038" Nov 01 15:39:45.385260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:45.385184 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:47.043499 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:47.043466 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:47.854851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:47.854817 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:48.019191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:48.019156 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:49.302770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:49.302735 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:49.726889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:49.726804 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2"] Nov 01 15:39:49.727071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:49.726917 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:49.727166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:49.727048 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" containerID="cri-o://19cd2acaddd379e738c40831104d738bb5a6011d27306134acb1a62b4ec81750" gracePeriod=30 Nov 01 15:39:49.899750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:49.899720 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:50.252336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.252302 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:50.827572 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.827530 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f"] Nov 01 15:39:50.828409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.828386 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6c085104-f3de-4d3f-abc3-3be9c57585d3" containerName="revoke-keys" Nov 01 15:39:50.828409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.828409 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c085104-f3de-4d3f-abc3-3be9c57585d3" containerName="revoke-keys" Nov 01 15:39:50.828602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.828583 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c085104-f3de-4d3f-abc3-3be9c57585d3" containerName="revoke-keys" Nov 01 15:39:50.833735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.833713 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:50.888808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.888776 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f"] Nov 01 15:39:50.933706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.933677 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:39:50.933861 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:50.933737 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:39:51.013273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.013210 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.013483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.013311 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.013568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.013541 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6wz\" (UniqueName: \"kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.114454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.114372 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.114454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.114444 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.114726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.114591 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6wz\" (UniqueName: \"kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.115075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.115052 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.116858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.116833 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.174164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.174132 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6wz\" (UniqueName: \"kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz\") pod \"maas-api-revoke-sub-2bde8715-cfm8f\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.334892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.334860 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:51.448150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.448075 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:51.690118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.690084 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:51.827259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.827209 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f"] Nov 01 15:39:51.828475 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:51.828447 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e554956_20a3_4455_a3a0_06765c21a8ac.slice/crio-ab56b5321ba487b17f21878b0871721d956ca9887e29f4626a6769ab168e2f76 WatchSource:0}: Error finding container ab56b5321ba487b17f21878b0871721d956ca9887e29f4626a6769ab168e2f76: Status 404 returned error can't find the container with id ab56b5321ba487b17f21878b0871721d956ca9887e29f4626a6769ab168e2f76 Nov 01 15:39:51.882024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:51.881987 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" event={"ID":"8e554956-20a3-4455-a3a0-06765c21a8ac","Type":"ContainerStarted","Data":"ab56b5321ba487b17f21878b0871721d956ca9887e29f4626a6769ab168e2f76"} Nov 01 15:39:52.891832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:52.891800 5473 generic.go:356] "Generic (PLEG): container finished" podID="8e554956-20a3-4455-a3a0-06765c21a8ac" containerID="741ad7b253d7151ebea909b7ffdba3e353a03a790a3fb86adecc72d901380054" exitCode=0 Nov 01 15:39:52.892259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:52.891902 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" event={"ID":"8e554956-20a3-4455-a3a0-06765c21a8ac","Type":"ContainerDied","Data":"741ad7b253d7151ebea909b7ffdba3e353a03a790a3fb86adecc72d901380054"} Nov 01 15:39:53.567979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.567949 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv"] Nov 01 15:39:53.572068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.572050 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.627953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.627919 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv"] Nov 01 15:39:53.645352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.645319 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.645550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.645528 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.645613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.645592 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v4hl\" (UniqueName: \"kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.747114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.747068 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.747332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.747261 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v4hl\" (UniqueName: \"kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.747381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.747343 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.747908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.747886 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.749815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.749790 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.806687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.806634 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v4hl\" (UniqueName: \"kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl\") pod \"maas-api-revoke-sub-245b55db-sjdgv\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:53.886162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:53.886083 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:54.090807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.090775 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:54.152367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.152293 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz\" (UniqueName: \"kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz\") pod \"8e554956-20a3-4455-a3a0-06765c21a8ac\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " Nov 01 15:39:54.152367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.152354 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca\") pod \"8e554956-20a3-4455-a3a0-06765c21a8ac\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " Nov 01 15:39:54.152634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.152608 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca\") pod \"8e554956-20a3-4455-a3a0-06765c21a8ac\" (UID: \"8e554956-20a3-4455-a3a0-06765c21a8ac\") " Nov 01 15:39:54.152787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.152764 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca" pod "8e554956-20a3-4455-a3a0-06765c21a8ac" (UID: "8e554956-20a3-4455-a3a0-06765c21a8ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:54.153166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.153140 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:54.154627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.154599 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz" pod "8e554956-20a3-4455-a3a0-06765c21a8ac" (UID: "8e554956-20a3-4455-a3a0-06765c21a8ac"). InnerVolumeSpecName "kube-api-access-5w6wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:54.154726 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.154640 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca" pod "8e554956-20a3-4455-a3a0-06765c21a8ac" (UID: "8e554956-20a3-4455-a3a0-06765c21a8ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:54.254657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.254625 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e554956-20a3-4455-a3a0-06765c21a8ac-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:54.254657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.254656 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6wz\" (UniqueName: \"kubernetes.io/projected/8e554956-20a3-4455-a3a0-06765c21a8ac-kube-api-access-5w6wz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:54.267521 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:39:54.267495 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de84e4a_7a88_480a_9685_670fb0ac47a9.slice/crio-885b73f83a4083773bfc8f5866fb8f6928a5c169c909923fe1e6cbd0050ed3a4 WatchSource:0}: Error finding container 885b73f83a4083773bfc8f5866fb8f6928a5c169c909923fe1e6cbd0050ed3a4: Status 404 returned error can't find the container with id 885b73f83a4083773bfc8f5866fb8f6928a5c169c909923fe1e6cbd0050ed3a4 Nov 01 15:39:54.267692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.267671 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv"] Nov 01 15:39:54.913286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.913248 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" event={"ID":"6de84e4a-7a88-480a-9685-670fb0ac47a9","Type":"ContainerStarted","Data":"703d417231ef142492283b2c957f392895f43a92f1528e0496fce5f58bf71134"} Nov 01 15:39:54.913466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.913322 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" event={"ID":"6de84e4a-7a88-480a-9685-670fb0ac47a9","Type":"ContainerStarted","Data":"885b73f83a4083773bfc8f5866fb8f6928a5c169c909923fe1e6cbd0050ed3a4"} Nov 01 15:39:54.915058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.915035 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" Nov 01 15:39:54.915196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.915082 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f" event={"ID":"8e554956-20a3-4455-a3a0-06765c21a8ac","Type":"ContainerDied","Data":"ab56b5321ba487b17f21878b0871721d956ca9887e29f4626a6769ab168e2f76"} Nov 01 15:39:54.915196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.915166 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab56b5321ba487b17f21878b0871721d956ca9887e29f4626a6769ab168e2f76" Nov 01 15:39:54.919446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.919422 5473 generic.go:356] "Generic (PLEG): container finished" podID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerID="19cd2acaddd379e738c40831104d738bb5a6011d27306134acb1a62b4ec81750" exitCode=0 Nov 01 15:39:54.919554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.919462 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" event={"ID":"3a028562-7f7b-4d29-8856-f191a1d6a30d","Type":"ContainerDied","Data":"19cd2acaddd379e738c40831104d738bb5a6011d27306134acb1a62b4ec81750"} Nov 01 15:39:54.919554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:54.919506 5473 scope.go:122] "RemoveContainer" containerID="173b617f94c566fb08e5d055833b40bf86969f708185e760cbc38474ce607e9f" Nov 01 15:39:55.001558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.001501 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:39:55.030801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.030752 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" podStartSLOduration=2.030737318 podStartE2EDuration="2.030737318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:39:55.02738122 +0000 UTC m=+4934.085533070" watchObservedRunningTime="2026-11-01 15:39:55.030737318 +0000 UTC m=+4934.088889168" Nov 01 15:39:55.064908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.064872 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.064927 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.064949 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.064970 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.064989 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065031 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065123 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065155 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065302 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data\") pod \"3a028562-7f7b-4d29-8856-f191a1d6a30d\" (UID: \"3a028562-7f7b-4d29-8856-f191a1d6a30d\") " Nov 01 15:39:55.065380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065315 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:55.065380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065372 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:55.065661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065438 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:55.065661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065590 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:55.065780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.065680 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:55.066091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.066063 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.066255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.066094 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a028562-7f7b-4d29-8856-f191a1d6a30d-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.066255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.066111 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.066255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.066126 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.066255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.066139 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.067452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.067427 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:39:55.067580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.067459 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:55.067580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.067478 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:39:55.067580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.067528 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn" pod "3a028562-7f7b-4d29-8856-f191a1d6a30d" (UID: "3a028562-7f7b-4d29-8856-f191a1d6a30d"). InnerVolumeSpecName "kube-api-access-sgxvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:55.167603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.167508 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgxvn\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-kube-api-access-sgxvn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.167603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.167544 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.167603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.167559 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.167603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.167574 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a028562-7f7b-4d29-8856-f191a1d6a30d-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:55.179955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.179925 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc"] Nov 01 15:39:55.197682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.197651 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7"] Nov 01 15:39:55.325512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.325478 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:55.929124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.929094 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" Nov 01 15:39:55.929124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.929113 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2" event={"ID":"3a028562-7f7b-4d29-8856-f191a1d6a30d","Type":"ContainerDied","Data":"6bbcd24f6f7ec67a1a878e87008dda1e8c34a0f0c04f6b6c0cba0471dd2b18ba"} Nov 01 15:39:55.929356 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.929197 5473 scope.go:122] "RemoveContainer" containerID="19cd2acaddd379e738c40831104d738bb5a6011d27306134acb1a62b4ec81750" Nov 01 15:39:55.931280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.931251 5473 generic.go:356] "Generic (PLEG): container finished" podID="6de84e4a-7a88-480a-9685-670fb0ac47a9" containerID="703d417231ef142492283b2c957f392895f43a92f1528e0496fce5f58bf71134" exitCode=0 Nov 01 15:39:55.931528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:55.931309 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" event={"ID":"6de84e4a-7a88-480a-9685-670fb0ac47a9","Type":"ContainerDied","Data":"703d417231ef142492283b2c957f392895f43a92f1528e0496fce5f58bf71134"} Nov 01 15:39:56.091112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:56.091029 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:39:56.098899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:56.098868 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2"] Nov 01 15:39:56.148698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:56.148660 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-974865-openshift-default-796bcb9c7f-2gwp2"] Nov 01 15:39:57.083070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.083043 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" path="/var/lib/kubelet/pods/3a028562-7f7b-4d29-8856-f191a1d6a30d/volumes" Nov 01 15:39:57.099579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.099551 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:57.195885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.195793 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca\") pod \"6de84e4a-7a88-480a-9685-670fb0ac47a9\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " Nov 01 15:39:57.196062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.195906 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca\") pod \"6de84e4a-7a88-480a-9685-670fb0ac47a9\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " Nov 01 15:39:57.196062 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.196023 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl\" (UniqueName: \"kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl\") pod \"6de84e4a-7a88-480a-9685-670fb0ac47a9\" (UID: \"6de84e4a-7a88-480a-9685-670fb0ac47a9\") " Nov 01 15:39:57.196301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.196277 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca" pod "6de84e4a-7a88-480a-9685-670fb0ac47a9" (UID: "6de84e4a-7a88-480a-9685-670fb0ac47a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:39:57.196776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.196732 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:57.198283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.198259 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl" pod "6de84e4a-7a88-480a-9685-670fb0ac47a9" (UID: "6de84e4a-7a88-480a-9685-670fb0ac47a9"). InnerVolumeSpecName "kube-api-access-2v4hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:39:57.198385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.198263 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca" pod "6de84e4a-7a88-480a-9685-670fb0ac47a9" (UID: "6de84e4a-7a88-480a-9685-670fb0ac47a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:39:57.297757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.297725 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v4hl\" (UniqueName: \"kubernetes.io/projected/6de84e4a-7a88-480a-9685-670fb0ac47a9-kube-api-access-2v4hl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:57.297757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.297754 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de84e4a-7a88-480a-9685-670fb0ac47a9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:39:57.950195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.950162 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" Nov 01 15:39:57.950413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.950160 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv" event={"ID":"6de84e4a-7a88-480a-9685-670fb0ac47a9","Type":"ContainerDied","Data":"885b73f83a4083773bfc8f5866fb8f6928a5c169c909923fe1e6cbd0050ed3a4"} Nov 01 15:39:57.950413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:57.950297 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="885b73f83a4083773bfc8f5866fb8f6928a5c169c909923fe1e6cbd0050ed3a4" Nov 01 15:39:58.947878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:58.947847 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7"] Nov 01 15:39:58.974166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:39:58.974136 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:00.488025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:00.487983 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:01.108855 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:01.108819 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:40:01.438720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:01.438644 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:02.363353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:02.363318 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:02.531778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:02.531741 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:04.935197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:04.935121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:06.585294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585261 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz"] Nov 01 15:40:06.585911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585897 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" Nov 01 15:40:06.585954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585912 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" Nov 01 15:40:06.585954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585935 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6de84e4a-7a88-480a-9685-670fb0ac47a9" containerName="revoke-keys" Nov 01 15:40:06.585954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585941 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de84e4a-7a88-480a-9685-670fb0ac47a9" containerName="revoke-keys" Nov 01 15:40:06.585954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585948 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8e554956-20a3-4455-a3a0-06765c21a8ac" containerName="revoke-keys" Nov 01 15:40:06.585954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.585954 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e554956-20a3-4455-a3a0-06765c21a8ac" containerName="revoke-keys" Nov 01 15:40:06.586117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.586050 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" Nov 01 15:40:06.586117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.586062 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" Nov 01 15:40:06.586117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.586069 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de84e4a-7a88-480a-9685-670fb0ac47a9" containerName="revoke-keys" Nov 01 15:40:06.586117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.586075 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e554956-20a3-4455-a3a0-06765c21a8ac" containerName="revoke-keys" Nov 01 15:40:06.590247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.590205 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.646111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.646077 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz"] Nov 01 15:40:06.691257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.691209 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:40:06.691478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.691457 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:40:06.723719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.723685 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.723909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.723726 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.723909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.723860 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnvj\" (UniqueName: \"kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.824995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.824956 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnvj\" (UniqueName: \"kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.825190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.825170 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.825319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.825207 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.825888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.825866 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.827588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.827566 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.882437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.882361 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnvj\" (UniqueName: \"kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj\") pod \"maas-api-revoke-sub-cb4f15bc-np2tz\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:06.905745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:06.905716 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:07.290711 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:40:07.290678 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08493723_38e8_4c45_8b92_ba2ec8d542cb.slice/crio-21e49a3be9beee1a77c165e76fde9db7e25d93c94821798b88fb131997daa184 WatchSource:0}: Error finding container 21e49a3be9beee1a77c165e76fde9db7e25d93c94821798b88fb131997daa184: Status 404 returned error can't find the container with id 21e49a3be9beee1a77c165e76fde9db7e25d93c94821798b88fb131997daa184 Nov 01 15:40:07.291145 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:07.291118 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz"] Nov 01 15:40:07.970693 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:07.970602 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz"] Nov 01 15:40:08.037701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.037655 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" event={"ID":"08493723-38e8-4c45-8b92-ba2ec8d542cb","Type":"ContainerStarted","Data":"fbb129ab9b9ca0695dd7a528eadbd199ab3996990b24b0ab65c28b3e77478682"} Nov 01 15:40:08.037887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.037731 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" event={"ID":"08493723-38e8-4c45-8b92-ba2ec8d542cb","Type":"ContainerStarted","Data":"21e49a3be9beee1a77c165e76fde9db7e25d93c94821798b88fb131997daa184"} Nov 01 15:40:08.076868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.076834 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb"] Nov 01 15:40:08.104098 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.104060 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:08.337928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.337876 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" podStartSLOduration=2.33786122 podStartE2EDuration="2.33786122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:40:08.333594455 +0000 UTC m=+4947.391746305" watchObservedRunningTime="2026-11-01 15:40:08.33786122 +0000 UTC m=+4947.396013070" Nov 01 15:40:08.368754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.368724 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:08.530149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:08.530115 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:09.047621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:09.047590 5473 generic.go:356] "Generic (PLEG): container finished" podID="08493723-38e8-4c45-8b92-ba2ec8d542cb" containerID="fbb129ab9b9ca0695dd7a528eadbd199ab3996990b24b0ab65c28b3e77478682" exitCode=0 Nov 01 15:40:09.048123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:09.047671 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" event={"ID":"08493723-38e8-4c45-8b92-ba2ec8d542cb","Type":"ContainerDied","Data":"fbb129ab9b9ca0695dd7a528eadbd199ab3996990b24b0ab65c28b3e77478682"} Nov 01 15:40:09.307682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:09.307592 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:09.469492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:09.469448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:09.660465 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:09.660385 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:10.057328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.057303 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:10.201714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.201676 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" containerID="cri-o://d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5" gracePeriod=45 Nov 01 15:40:10.218674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.218632 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" containerID="cri-o://df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec" gracePeriod=45 Nov 01 15:40:10.297656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.297624 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 01 15:40:10.357287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.357207 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:10.475661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.474662 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca\") pod \"08493723-38e8-4c45-8b92-ba2ec8d542cb\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " Nov 01 15:40:10.475661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.474759 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj\" (UniqueName: \"kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj\") pod \"08493723-38e8-4c45-8b92-ba2ec8d542cb\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " Nov 01 15:40:10.475661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.474990 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca\") pod \"08493723-38e8-4c45-8b92-ba2ec8d542cb\" (UID: \"08493723-38e8-4c45-8b92-ba2ec8d542cb\") " Nov 01 15:40:10.486311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.486275 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca" pod "08493723-38e8-4c45-8b92-ba2ec8d542cb" (UID: "08493723-38e8-4c45-8b92-ba2ec8d542cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:10.494814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.494758 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca" pod "08493723-38e8-4c45-8b92-ba2ec8d542cb" (UID: "08493723-38e8-4c45-8b92-ba2ec8d542cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:10.494814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.494786 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj" pod "08493723-38e8-4c45-8b92-ba2ec8d542cb" (UID: "08493723-38e8-4c45-8b92-ba2ec8d542cb"). InnerVolumeSpecName "kube-api-access-qtnvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:10.507760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.507730 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 15:40:10.513993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.513970 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 15:40:10.577330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.577196 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.577330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.577279 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08493723-38e8-4c45-8b92-ba2ec8d542cb-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.577330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.577297 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtnvj\" (UniqueName: \"kubernetes.io/projected/08493723-38e8-4c45-8b92-ba2ec8d542cb-kube-api-access-qtnvj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.678028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.677996 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache\") pod \"89542591-7b5b-47d1-aa56-fed62b5a2290\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " Nov 01 15:40:10.678252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678054 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir\") pod \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " Nov 01 15:40:10.678252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678109 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6\" (UniqueName: \"kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6\") pod \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " Nov 01 15:40:10.678252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678149 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9\" (UniqueName: \"kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9\") pod \"89542591-7b5b-47d1-aa56-fed62b5a2290\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " Nov 01 15:40:10.678252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678202 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home\") pod \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678263 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs\" (UniqueName: \"kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs\") pod \"89542591-7b5b-47d1-aa56-fed62b5a2290\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678304 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs\") pod \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678324 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir" pod "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" (UID: "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678335 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache" pod "89542591-7b5b-47d1-aa56-fed62b5a2290" (UID: "89542591-7b5b-47d1-aa56-fed62b5a2290"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678335 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir\") pod \"89542591-7b5b-47d1-aa56-fed62b5a2290\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678389 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home\") pod \"89542591-7b5b-47d1-aa56-fed62b5a2290\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " Nov 01 15:40:10.678493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678457 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm\") pod \"89542591-7b5b-47d1-aa56-fed62b5a2290\" (UID: \"89542591-7b5b-47d1-aa56-fed62b5a2290\") " Nov 01 15:40:10.678866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678571 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir" pod "89542591-7b5b-47d1-aa56-fed62b5a2290" (UID: "89542591-7b5b-47d1-aa56-fed62b5a2290"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.678866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678646 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm\") pod \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " Nov 01 15:40:10.678866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678704 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache\") pod \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\" (UID: \"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7\") " Nov 01 15:40:10.678866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678728 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home" pod "89542591-7b5b-47d1-aa56-fed62b5a2290" (UID: "89542591-7b5b-47d1-aa56-fed62b5a2290"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.679096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.678875 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home" pod "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" (UID: "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.679250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679155 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache" pod "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" (UID: "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.679360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679337 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.679430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679394 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.679495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679437 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.679495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679461 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.679495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679475 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.679495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.679490 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.680842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.680811 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm" pod "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" (UID: "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.680958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.680884 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm" pod "89542591-7b5b-47d1-aa56-fed62b5a2290" (UID: "89542591-7b5b-47d1-aa56-fed62b5a2290"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:10.681051 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.681031 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs" pod "89542591-7b5b-47d1-aa56-fed62b5a2290" (UID: "89542591-7b5b-47d1-aa56-fed62b5a2290"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:10.681108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.681045 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9" pod "89542591-7b5b-47d1-aa56-fed62b5a2290" (UID: "89542591-7b5b-47d1-aa56-fed62b5a2290"). InnerVolumeSpecName "kube-api-access-hhhv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:10.681703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.681684 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs" pod "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" (UID: "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:10.682301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.682284 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6" pod "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" (UID: "4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7"). InnerVolumeSpecName "kube-api-access-82lh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:10.780709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.780679 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.780709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.780706 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89542591-7b5b-47d1-aa56-fed62b5a2290-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.780709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.780716 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.780709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.780725 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82lh6\" (UniqueName: \"kubernetes.io/projected/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7-kube-api-access-82lh6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.780984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.780738 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhhv9\" (UniqueName: \"kubernetes.io/projected/89542591-7b5b-47d1-aa56-fed62b5a2290-kube-api-access-hhhv9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:10.780984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:10.780750 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89542591-7b5b-47d1-aa56-fed62b5a2290-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:11.069642 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.069601 5473 generic.go:356] "Generic (PLEG): container finished" podID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerID="df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec" exitCode=0 Nov 01 15:40:11.070163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.069671 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" Nov 01 15:40:11.070163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.069687 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" event={"ID":"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7","Type":"ContainerDied","Data":"df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec"} Nov 01 15:40:11.070163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.069738 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7" event={"ID":"4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7","Type":"ContainerDied","Data":"88c9fedaac3ff9922974dbf30bd82a7d2ade4b0cf8ebd4546556c52b1ef4007a"} Nov 01 15:40:11.070163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.069763 5473 scope.go:122] "RemoveContainer" containerID="df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec" Nov 01 15:40:11.071463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.071445 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" Nov 01 15:40:11.071589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.071494 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz" event={"ID":"08493723-38e8-4c45-8b92-ba2ec8d542cb","Type":"ContainerDied","Data":"21e49a3be9beee1a77c165e76fde9db7e25d93c94821798b88fb131997daa184"} Nov 01 15:40:11.071678 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.071596 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21e49a3be9beee1a77c165e76fde9db7e25d93c94821798b88fb131997daa184" Nov 01 15:40:11.073067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.073043 5473 generic.go:356] "Generic (PLEG): container finished" podID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerID="d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5" exitCode=0 Nov 01 15:40:11.073298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.073158 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" event={"ID":"89542591-7b5b-47d1-aa56-fed62b5a2290","Type":"ContainerDied","Data":"d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5"} Nov 01 15:40:11.073298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.073185 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" event={"ID":"89542591-7b5b-47d1-aa56-fed62b5a2290","Type":"ContainerDied","Data":"899baaba9d8ccd5d714107fc8aa193b20d6a89c4c88e4aabebc56f3dd12c1640"} Nov 01 15:40:11.073298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.073260 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc" Nov 01 15:40:11.088251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.088199 5473 scope.go:122] "RemoveContainer" containerID="df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec" Nov 01 15:40:11.088666 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:11.088566 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec\": container with ID starting with df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec not found: ID does not exist" containerID="df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec" Nov 01 15:40:11.088666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.088599 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec"} err="failed to get container status \"df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec\": rpc error: code = NotFound desc = could not find container \"df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec\": container with ID starting with df5360c0735b8b3707cc2fb58c1b150877b8b38a8ecc3a3b6276f702b4ef07ec not found: ID does not exist" Nov 01 15:40:11.088666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.088618 5473 scope.go:122] "RemoveContainer" containerID="d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5" Nov 01 15:40:11.099438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.099421 5473 scope.go:122] "RemoveContainer" containerID="d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5" Nov 01 15:40:11.099718 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:11.099699 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5\": container with ID starting with d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5 not found: ID does not exist" containerID="d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5" Nov 01 15:40:11.099782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.099725 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5"} err="failed to get container status \"d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5\": rpc error: code = NotFound desc = could not find container \"d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5\": container with ID starting with d9e43f698ec6b75e640e69ac7e20c7e812aa0b45a17967f02a7d60661257e2d5 not found: ID does not exist" Nov 01 15:40:11.239816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.239780 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc"] Nov 01 15:40:11.290039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.290003 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/distinct-2-w0-1f8e2c-kserve-64ff4d66b5-6gmtc"] Nov 01 15:40:11.451229 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.451128 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7"] Nov 01 15:40:11.455185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:11.455159 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1f8e2c/premium-w0-1f8e2c-kserve-7bd894b4c-qc9p7"] Nov 01 15:40:12.880783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:12.880750 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd"] Nov 01 15:40:12.937627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:12.937596 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:12.975403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:12.971878 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerName="payload-processing" containerID="cri-o://d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c" gracePeriod=25 Nov 01 15:40:13.077676 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.077636 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" containerName="payload-pre-processing" containerID="cri-o://01130abfafcc34fb2467d26467d26395d841786c19de8491283563f5ce768259" gracePeriod=25 Nov 01 15:40:13.091558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.088377 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" path="/var/lib/kubelet/pods/4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7/volumes" Nov 01 15:40:13.091558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.089132 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" path="/var/lib/kubelet/pods/89542591-7b5b-47d1-aa56-fed62b5a2290/volumes" Nov 01 15:40:13.643990 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.643954 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:13.972586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.972491 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" podUID="4c88bb0b-4801-42ae-9006-6c1006a06c35" containerName="main" containerID="cri-o://7198851b84cb62df0eac7de66febadac38a6e5088e6d374cdfaf8da295a39b26" gracePeriod=45 Nov 01 15:40:13.977852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.977820 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78"] Nov 01 15:40:13.978802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.978776 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" Nov 01 15:40:13.979635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979614 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" Nov 01 15:40:13.979772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979759 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" Nov 01 15:40:13.979862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979846 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a028562-7f7b-4d29-8856-f191a1d6a30d" containerName="istio-proxy" Nov 01 15:40:13.979933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979889 5473 cpu_manager.go:414] "containerMap: removing container" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" Nov 01 15:40:13.979933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979898 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" Nov 01 15:40:13.980055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979953 5473 cpu_manager.go:414] "containerMap: removing container" podUID="08493723-38e8-4c45-8b92-ba2ec8d542cb" containerName="revoke-keys" Nov 01 15:40:13.980055 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.979962 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08493723-38e8-4c45-8b92-ba2ec8d542cb" containerName="revoke-keys" Nov 01 15:40:13.980175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.980160 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="08493723-38e8-4c45-8b92-ba2ec8d542cb" containerName="revoke-keys" Nov 01 15:40:13.980281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.980189 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb8da5b-2f76-4b39-b0b2-0dbdf72f83a7" containerName="main" Nov 01 15:40:13.980281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.980206 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="89542591-7b5b-47d1-aa56-fed62b5a2290" containerName="main" Nov 01 15:40:13.985587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:13.985564 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.063183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.063147 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:40:14.063445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.063259 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:40:14.067247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.067201 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78"] Nov 01 15:40:14.122396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.122356 5473 generic.go:356] "Generic (PLEG): container finished" podID="4c88bb0b-4801-42ae-9006-6c1006a06c35" containerID="7198851b84cb62df0eac7de66febadac38a6e5088e6d374cdfaf8da295a39b26" exitCode=0 Nov 01 15:40:14.122579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.122536 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" event={"ID":"4c88bb0b-4801-42ae-9006-6c1006a06c35","Type":"ContainerDied","Data":"7198851b84cb62df0eac7de66febadac38a6e5088e6d374cdfaf8da295a39b26"} Nov 01 15:40:14.127072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.127042 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.127255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.127077 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.127255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.127105 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.228903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.228871 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.229169 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.229147 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.229249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.229192 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.229816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.229793 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.231790 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.231765 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.255669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.255636 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:40:14.286040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.286011 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj\") pod \"maas-api-revoke-sub-550acb42-sjc78\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.329953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.329921 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm\") pod \"4c88bb0b-4801-42ae-9006-6c1006a06c35\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " Nov 01 15:40:14.330168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.329982 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home\") pod \"4c88bb0b-4801-42ae-9006-6c1006a06c35\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " Nov 01 15:40:14.330168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330041 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir\") pod \"4c88bb0b-4801-42ae-9006-6c1006a06c35\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " Nov 01 15:40:14.330168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330110 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache\") pod \"4c88bb0b-4801-42ae-9006-6c1006a06c35\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " Nov 01 15:40:14.330168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330136 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs\") pod \"4c88bb0b-4801-42ae-9006-6c1006a06c35\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " Nov 01 15:40:14.330438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330172 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp\" (UniqueName: \"kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp\") pod \"4c88bb0b-4801-42ae-9006-6c1006a06c35\" (UID: \"4c88bb0b-4801-42ae-9006-6c1006a06c35\") " Nov 01 15:40:14.330438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330323 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache" pod "4c88bb0b-4801-42ae-9006-6c1006a06c35" (UID: "4c88bb0b-4801-42ae-9006-6c1006a06c35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:14.330438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330336 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home" pod "4c88bb0b-4801-42ae-9006-6c1006a06c35" (UID: "4c88bb0b-4801-42ae-9006-6c1006a06c35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:14.330438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330388 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir" pod "4c88bb0b-4801-42ae-9006-6c1006a06c35" (UID: "4c88bb0b-4801-42ae-9006-6c1006a06c35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:14.330765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330742 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:14.330765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330764 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:14.330765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.330774 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:14.332376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.332347 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm" pod "4c88bb0b-4801-42ae-9006-6c1006a06c35" (UID: "4c88bb0b-4801-42ae-9006-6c1006a06c35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:14.332479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.332388 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs" pod "4c88bb0b-4801-42ae-9006-6c1006a06c35" (UID: "4c88bb0b-4801-42ae-9006-6c1006a06c35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:14.332479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.332388 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp" pod "4c88bb0b-4801-42ae-9006-6c1006a06c35" (UID: "4c88bb0b-4801-42ae-9006-6c1006a06c35"). InnerVolumeSpecName "kube-api-access-6b4bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:14.381110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.381074 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:14.439265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.433888 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c88bb0b-4801-42ae-9006-6c1006a06c35-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:14.439265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.433929 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c88bb0b-4801-42ae-9006-6c1006a06c35-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:14.439265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.433945 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b4bp\" (UniqueName: \"kubernetes.io/projected/4c88bb0b-4801-42ae-9006-6c1006a06c35-kube-api-access-6b4bp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:14.541660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.541628 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:14.805897 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:40:14.805864 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9030abb_d4c8_4da4_9460_9ae73221a9eb.slice/crio-e0915a59a87ee04980d2019dede1b21c7a5af80989fa6bfbe4bd0faef918fd00 WatchSource:0}: Error finding container e0915a59a87ee04980d2019dede1b21c7a5af80989fa6bfbe4bd0faef918fd00: Status 404 returned error can't find the container with id e0915a59a87ee04980d2019dede1b21c7a5af80989fa6bfbe4bd0faef918fd00 Nov 01 15:40:14.806518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:14.806494 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78"] Nov 01 15:40:15.132748 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.132719 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" Nov 01 15:40:15.133162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.132809 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" event={"ID":"4c88bb0b-4801-42ae-9006-6c1006a06c35","Type":"ContainerDied","Data":"aad81375903544426966d73477d342ad2a72aee997ea61a064ced127e450b897"} Nov 01 15:40:15.133162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.132862 5473 scope.go:122] "RemoveContainer" containerID="7198851b84cb62df0eac7de66febadac38a6e5088e6d374cdfaf8da295a39b26" Nov 01 15:40:15.134556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.134352 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" event={"ID":"f9030abb-d4c8-4da4-9460-9ae73221a9eb","Type":"ContainerStarted","Data":"e0915a59a87ee04980d2019dede1b21c7a5af80989fa6bfbe4bd0faef918fd00"} Nov 01 15:40:15.205380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.205337 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7" podUID="4c88bb0b-4801-42ae-9006-6c1006a06c35" containerName="main" probeResult="failure" output="Get \"https://10.134.1.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 15:40:15.300711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.300668 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7"] Nov 01 15:40:15.304300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.304277 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ac11c/rate-test-model-1ac11c-kserve-556c8bd789-4gwx7"] Nov 01 15:40:15.398901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:15.398825 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:16.139088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:16.139057 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:16.145133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:16.145104 5473 generic.go:356] "Generic (PLEG): container finished" podID="f9030abb-d4c8-4da4-9460-9ae73221a9eb" containerID="1a31d5ed39a71c82e51f857037304718548b5d0e3c9b58c91cd9d9da7ff1c26e" exitCode=0 Nov 01 15:40:16.145294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:16.145144 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" event={"ID":"f9030abb-d4c8-4da4-9460-9ae73221a9eb","Type":"ContainerDied","Data":"1a31d5ed39a71c82e51f857037304718548b5d0e3c9b58c91cd9d9da7ff1c26e"} Nov 01 15:40:16.675806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:16.675774 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq"] Nov 01 15:40:16.783140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:16.783108 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2"] Nov 01 15:40:17.045230 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.045189 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:17.087016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.086979 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c88bb0b-4801-42ae-9006-6c1006a06c35" path="/var/lib/kubelet/pods/4c88bb0b-4801-42ae-9006-6c1006a06c35/volumes" Nov 01 15:40:17.195652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.195618 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:17.324259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.324230 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:17.477750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.477710 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca\") pod \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " Nov 01 15:40:17.477950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.477910 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca\") pod \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " Nov 01 15:40:17.477950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.477945 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj\") pod \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\" (UID: \"f9030abb-d4c8-4da4-9460-9ae73221a9eb\") " Nov 01 15:40:17.478210 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.478178 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca" pod "f9030abb-d4c8-4da4-9460-9ae73221a9eb" (UID: "f9030abb-d4c8-4da4-9460-9ae73221a9eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:17.478490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.478469 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:17.480353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.480322 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj" pod "f9030abb-d4c8-4da4-9460-9ae73221a9eb" (UID: "f9030abb-d4c8-4da4-9460-9ae73221a9eb"). InnerVolumeSpecName "kube-api-access-qvmcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:17.480489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.480456 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca" pod "f9030abb-d4c8-4da4-9460-9ae73221a9eb" (UID: "f9030abb-d4c8-4da4-9460-9ae73221a9eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:17.579277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.579191 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9030abb-d4c8-4da4-9460-9ae73221a9eb-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:17.579277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:17.579258 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmcj\" (UniqueName: \"kubernetes.io/projected/f9030abb-d4c8-4da4-9460-9ae73221a9eb-kube-api-access-qvmcj\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:18.161532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:18.161492 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" Nov 01 15:40:18.161532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:18.161513 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78" event={"ID":"f9030abb-d4c8-4da4-9460-9ae73221a9eb","Type":"ContainerDied","Data":"e0915a59a87ee04980d2019dede1b21c7a5af80989fa6bfbe4bd0faef918fd00"} Nov 01 15:40:18.161780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:18.161570 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0915a59a87ee04980d2019dede1b21c7a5af80989fa6bfbe4bd0faef918fd00" Nov 01 15:40:19.070415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:19.070380 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:19.070817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:19.070384 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:20.083904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:20.083868 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:20.243880 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:20.243841 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:21.553881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:21.553838 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:21.677013 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:21.676969 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerName="payload-processing" containerID="cri-o://ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4" gracePeriod=25 Nov 01 15:40:21.784012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:21.783966 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" containerName="payload-pre-processing" containerID="cri-o://3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f" gracePeriod=25 Nov 01 15:40:22.585842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:22.585806 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:22.786258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:22.786193 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:23.837796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:23.837756 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:23.838202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:23.837854 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:24.021598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:24.021560 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:24.430409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:24.430374 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:24.476231 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:24.476190 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt"] Nov 01 15:40:24.476474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:24.476451 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" containerID="cri-o://1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3" gracePeriod=30 Nov 01 15:40:24.605185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:24.605013 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:24.928863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:24.928830 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:25.451347 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.451310 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:25.477197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.477163 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj"] Nov 01 15:40:25.477844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.477827 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f9030abb-d4c8-4da4-9460-9ae73221a9eb" containerName="revoke-keys" Nov 01 15:40:25.477844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.477843 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9030abb-d4c8-4da4-9460-9ae73221a9eb" containerName="revoke-keys" Nov 01 15:40:25.477978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.477872 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4c88bb0b-4801-42ae-9006-6c1006a06c35" containerName="main" Nov 01 15:40:25.477978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.477878 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c88bb0b-4801-42ae-9006-6c1006a06c35" containerName="main" Nov 01 15:40:25.478047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.477998 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9030abb-d4c8-4da4-9460-9ae73221a9eb" containerName="revoke-keys" Nov 01 15:40:25.478047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.478013 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c88bb0b-4801-42ae-9006-6c1006a06c35" containerName="main" Nov 01 15:40:25.481804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.481786 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.535651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.535041 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:40:25.535651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.535168 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:40:25.539529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.539500 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj"] Nov 01 15:40:25.586809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.586778 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pv7d\" (UniqueName: \"kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.586960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.586817 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.586960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.586928 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.688660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.688625 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.688912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.688695 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.688995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.688911 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pv7d\" (UniqueName: \"kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.689534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.689502 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.709651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.707240 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.748405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.748374 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pv7d\" (UniqueName: \"kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d\") pod \"maas-api-revoke-sub-f509b530-v24xj\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:25.796740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:25.796698 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:26.183647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:26.183609 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj"] Nov 01 15:40:26.184090 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:40:26.183619 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486e25f6_eabc_4114_8d18_5ab19e6868f8.slice/crio-da10ce628603d79ecd590861e93be614cdf0d2880b67779dbd5e45ddd14fa536 WatchSource:0}: Error finding container da10ce628603d79ecd590861e93be614cdf0d2880b67779dbd5e45ddd14fa536: Status 404 returned error can't find the container with id da10ce628603d79ecd590861e93be614cdf0d2880b67779dbd5e45ddd14fa536 Nov 01 15:40:26.227029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:26.226997 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" event={"ID":"486e25f6-eabc-4114-8d18-5ab19e6868f8","Type":"ContainerStarted","Data":"da10ce628603d79ecd590861e93be614cdf0d2880b67779dbd5e45ddd14fa536"} Nov 01 15:40:26.871677 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:26.871643 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:27.049019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:27.048977 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:27.217066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:27.216978 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:27.238195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:27.238156 5473 generic.go:356] "Generic (PLEG): container finished" podID="486e25f6-eabc-4114-8d18-5ab19e6868f8" containerID="7def863bfccf97e74d9e4538e534e4da9a0de6994304e55ed26570aecb97f082" exitCode=0 Nov 01 15:40:27.238426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:27.238244 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" event={"ID":"486e25f6-eabc-4114-8d18-5ab19e6868f8","Type":"ContainerDied","Data":"7def863bfccf97e74d9e4538e534e4da9a0de6994304e55ed26570aecb97f082"} Nov 01 15:40:27.417797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:27.417763 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:27.902017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:27.901965 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" podUID="7e9ccfa6-26bc-42e7-82da-7a599002ddd2" containerName="main" containerID="cri-o://929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305" gracePeriod=45 Nov 01 15:40:28.163749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.163684 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:40:28.217493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217457 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir\") pod \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217538 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx\" (UniqueName: \"kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx\") pod \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217568 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache\") pod \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217594 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home\") pod \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217677 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm\") pod \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217703 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir" pod "7e9ccfa6-26bc-42e7-82da-7a599002ddd2" (UID: "7e9ccfa6-26bc-42e7-82da-7a599002ddd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217738 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs\") pod \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\" (UID: \"7e9ccfa6-26bc-42e7-82da-7a599002ddd2\") " Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217895 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache" pod "7e9ccfa6-26bc-42e7-82da-7a599002ddd2" (UID: "7e9ccfa6-26bc-42e7-82da-7a599002ddd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:28.217995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.217949 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home" pod "7e9ccfa6-26bc-42e7-82da-7a599002ddd2" (UID: "7e9ccfa6-26bc-42e7-82da-7a599002ddd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:28.218530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.218129 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.218530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.218142 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.218530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.218154 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.219943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.219913 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs" pod "7e9ccfa6-26bc-42e7-82da-7a599002ddd2" (UID: "7e9ccfa6-26bc-42e7-82da-7a599002ddd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:28.220075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.219919 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm" pod "7e9ccfa6-26bc-42e7-82da-7a599002ddd2" (UID: "7e9ccfa6-26bc-42e7-82da-7a599002ddd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:28.220075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.220017 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx" pod "7e9ccfa6-26bc-42e7-82da-7a599002ddd2" (UID: "7e9ccfa6-26bc-42e7-82da-7a599002ddd2"). InnerVolumeSpecName "kube-api-access-88qpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:28.246635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.246604 5473 generic.go:356] "Generic (PLEG): container finished" podID="7e9ccfa6-26bc-42e7-82da-7a599002ddd2" containerID="929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305" exitCode=0 Nov 01 15:40:28.246830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.246652 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" Nov 01 15:40:28.246830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.246688 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" event={"ID":"7e9ccfa6-26bc-42e7-82da-7a599002ddd2","Type":"ContainerDied","Data":"929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305"} Nov 01 15:40:28.246830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.246733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd" event={"ID":"7e9ccfa6-26bc-42e7-82da-7a599002ddd2","Type":"ContainerDied","Data":"63625e9c33e2712954eb1475c47842ab1be65d51e2b50e68972e4c365a33cf42"} Nov 01 15:40:28.246830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.246752 5473 scope.go:122] "RemoveContainer" containerID="929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305" Nov 01 15:40:28.260515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.260490 5473 scope.go:122] "RemoveContainer" containerID="929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305" Nov 01 15:40:28.260840 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:28.260817 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305\": container with ID starting with 929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305 not found: ID does not exist" containerID="929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305" Nov 01 15:40:28.260930 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.260849 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305"} err="failed to get container status \"929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305\": rpc error: code = NotFound desc = could not find container \"929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305\": container with ID starting with 929acff41be18e741aecaca42c43227f590324bce8c3d5d9e9578c9dc905b305 not found: ID does not exist" Nov 01 15:40:28.319088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.319053 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.319088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.319087 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.319282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.319103 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qpx\" (UniqueName: \"kubernetes.io/projected/7e9ccfa6-26bc-42e7-82da-7a599002ddd2-kube-api-access-88qpx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.393144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.393106 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd"] Nov 01 15:40:28.396915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.396886 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-96165a/test-model-96165a-kserve-8fc6c468c-n4pzd"] Nov 01 15:40:28.404457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.404435 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:28.415785 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.415714 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:40:28.415915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.415777 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:40:28.420246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.420202 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca\") pod \"486e25f6-eabc-4114-8d18-5ab19e6868f8\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " Nov 01 15:40:28.420324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.420282 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca\") pod \"486e25f6-eabc-4114-8d18-5ab19e6868f8\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " Nov 01 15:40:28.420375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.420340 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d\" (UniqueName: \"kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d\") pod \"486e25f6-eabc-4114-8d18-5ab19e6868f8\" (UID: \"486e25f6-eabc-4114-8d18-5ab19e6868f8\") " Nov 01 15:40:28.420533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.420510 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca" pod "486e25f6-eabc-4114-8d18-5ab19e6868f8" (UID: "486e25f6-eabc-4114-8d18-5ab19e6868f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:28.420673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.420658 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.422449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.422426 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d" pod "486e25f6-eabc-4114-8d18-5ab19e6868f8" (UID: "486e25f6-eabc-4114-8d18-5ab19e6868f8"). InnerVolumeSpecName "kube-api-access-9pv7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:28.422449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.422435 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca" pod "486e25f6-eabc-4114-8d18-5ab19e6868f8" (UID: "486e25f6-eabc-4114-8d18-5ab19e6868f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:28.521678 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.521640 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/486e25f6-eabc-4114-8d18-5ab19e6868f8-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.521897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.521879 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pv7d\" (UniqueName: \"kubernetes.io/projected/486e25f6-eabc-4114-8d18-5ab19e6868f8-kube-api-access-9pv7d\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:28.761945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:28.761914 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:29.069812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.069714 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:29.069812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.069729 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:29.083831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.083799 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9ccfa6-26bc-42e7-82da-7a599002ddd2" path="/var/lib/kubelet/pods/7e9ccfa6-26bc-42e7-82da-7a599002ddd2/volumes" Nov 01 15:40:29.256574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.256545 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" Nov 01 15:40:29.256964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.256581 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj" event={"ID":"486e25f6-eabc-4114-8d18-5ab19e6868f8","Type":"ContainerDied","Data":"da10ce628603d79ecd590861e93be614cdf0d2880b67779dbd5e45ddd14fa536"} Nov 01 15:40:29.256964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.256630 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da10ce628603d79ecd590861e93be614cdf0d2880b67779dbd5e45ddd14fa536" Nov 01 15:40:29.263257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.263229 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:29.735224 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.735185 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:40:29.837564 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837533 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.837779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837587 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.837779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837625 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.837779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837683 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.837779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837716 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.838023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837782 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.838023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837871 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.838023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.837938 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.838023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838007 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token\") pod \"60c16441-be77-4a41-b90d-77b04df67172\" (UID: \"60c16441-be77-4a41-b90d-77b04df67172\") " Nov 01 15:40:29.838023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838005 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:29.838300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838054 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:29.838300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838179 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:29.838426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838305 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:29.838426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838320 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:29.838593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838568 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60c16441-be77-4a41-b90d-77b04df67172-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.838593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838593 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.838792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838604 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.838792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838614 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.838792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.838624 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.840377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.840344 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:40:29.840481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.840458 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "kube-api-access-mrtnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:29.840593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.840579 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:29.840682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.840669 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token" pod "60c16441-be77-4a41-b90d-77b04df67172" (UID: "60c16441-be77-4a41-b90d-77b04df67172"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:29.939893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.939852 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.939893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.939891 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60c16441-be77-4a41-b90d-77b04df67172-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.940115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.939908 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrtnb\" (UniqueName: \"kubernetes.io/projected/60c16441-be77-4a41-b90d-77b04df67172-kube-api-access-mrtnb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.940115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.939925 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60c16441-be77-4a41-b90d-77b04df67172-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:29.949602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:29.949571 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:30.104338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.104209 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:30.267196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.267166 5473 generic.go:356] "Generic (PLEG): container finished" podID="60c16441-be77-4a41-b90d-77b04df67172" containerID="1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3" exitCode=0 Nov 01 15:40:30.267655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.267208 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" Nov 01 15:40:30.267655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.267248 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" event={"ID":"60c16441-be77-4a41-b90d-77b04df67172","Type":"ContainerDied","Data":"1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3"} Nov 01 15:40:30.267655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.267305 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt" event={"ID":"60c16441-be77-4a41-b90d-77b04df67172","Type":"ContainerDied","Data":"353b1efcee94730495bb33a04b85d6c46d26d24b71f264b4d6aec467e6dd5889"} Nov 01 15:40:30.267655 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.267329 5473 scope.go:122] "RemoveContainer" containerID="1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3" Nov 01 15:40:30.280704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.280680 5473 scope.go:122] "RemoveContainer" containerID="e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640" Nov 01 15:40:30.291622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.291591 5473 scope.go:122] "RemoveContainer" containerID="1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3" Nov 01 15:40:30.291931 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:30.291908 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3\": container with ID starting with 1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3 not found: ID does not exist" containerID="1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3" Nov 01 15:40:30.291987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.291939 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3"} err="failed to get container status \"1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3\": rpc error: code = NotFound desc = could not find container \"1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3\": container with ID starting with 1e8c4bd467f4dbd3a70f314179eba9118dd5969d8bd7b3e41bfdd16540edbaf3 not found: ID does not exist" Nov 01 15:40:30.291987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.291957 5473 scope.go:122] "RemoveContainer" containerID="e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640" Nov 01 15:40:30.292257 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:30.292204 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640\": container with ID starting with e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640 not found: ID does not exist" containerID="e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640" Nov 01 15:40:30.292360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.292262 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640"} err="failed to get container status \"e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640\": rpc error: code = NotFound desc = could not find container \"e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640\": container with ID starting with e5a13c281ca76bdb2985b8ae1f2ed31d1fd833def60088f24969cd42bff81640 not found: ID does not exist" Nov 01 15:40:30.433639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.433549 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt"] Nov 01 15:40:30.437099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:30.437073 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b3f06c-openshift-default-74dc4d89f7-rtkgt"] Nov 01 15:40:31.083246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:31.083203 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c16441-be77-4a41-b90d-77b04df67172" path="/var/lib/kubelet/pods/60c16441-be77-4a41-b90d-77b04df67172/volumes" Nov 01 15:40:31.109625 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:31.109589 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:40:31.440719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:31.440627 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:31.604268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:31.604207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:31.898204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:31.898173 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:32.878841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:32.878808 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:33.217448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.217419 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 15:40:33.295822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.295790 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume\") pod \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " Nov 01 15:40:33.296046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.296026 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p\" (UniqueName: \"kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p\") pod \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\" (UID: \"6f0defb1-457c-44e2-a7c3-13c9bc08a042\") " Nov 01 15:40:33.296277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.296253 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume" pod "6f0defb1-457c-44e2-a7c3-13c9bc08a042" (UID: "6f0defb1-457c-44e2-a7c3-13c9bc08a042"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:33.296523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.296480 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0defb1-457c-44e2-a7c3-13c9bc08a042-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:33.298440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.298409 5473 generic.go:356] "Generic (PLEG): container finished" podID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerID="d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c" exitCode=0 Nov 01 15:40:33.298562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.298466 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" Nov 01 15:40:33.298562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.298508 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" event={"ID":"6f0defb1-457c-44e2-a7c3-13c9bc08a042","Type":"ContainerDied","Data":"d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c"} Nov 01 15:40:33.298700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.298587 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz" event={"ID":"6f0defb1-457c-44e2-a7c3-13c9bc08a042","Type":"ContainerDied","Data":"287ee99ec1e99cc5ea812a63d08d0ac856ad97291d530644b6343a8e682bb4f1"} Nov 01 15:40:33.298700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.298621 5473 scope.go:122] "RemoveContainer" containerID="d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c" Nov 01 15:40:33.298962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.298922 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p" pod "6f0defb1-457c-44e2-a7c3-13c9bc08a042" (UID: "6f0defb1-457c-44e2-a7c3-13c9bc08a042"). InnerVolumeSpecName "kube-api-access-c9v2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:33.300520 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.300496 5473 generic.go:356] "Generic (PLEG): container finished" podID="18da4621-7b38-450f-ab04-2e89039b46a2" containerID="01130abfafcc34fb2467d26467d26395d841786c19de8491283563f5ce768259" exitCode=0 Nov 01 15:40:33.300616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.300547 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" event={"ID":"18da4621-7b38-450f-ab04-2e89039b46a2","Type":"ContainerDied","Data":"01130abfafcc34fb2467d26467d26395d841786c19de8491283563f5ce768259"} Nov 01 15:40:33.309566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.309544 5473 scope.go:122] "RemoveContainer" containerID="d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c" Nov 01 15:40:33.309890 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:33.309870 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c\": container with ID starting with d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c not found: ID does not exist" containerID="d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c" Nov 01 15:40:33.309962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.309900 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c"} err="failed to get container status \"d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c\": rpc error: code = NotFound desc = could not find container \"d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c\": container with ID starting with d06e26e6484b3d5c2716ed4917a250dee9e27c3bf04cc3d87d8348e62257451c not found: ID does not exist" Nov 01 15:40:33.320449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.320428 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 15:40:33.397138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.397090 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume\") pod \"18da4621-7b38-450f-ab04-2e89039b46a2\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " Nov 01 15:40:33.397333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.397288 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp\" (UniqueName: \"kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp\") pod \"18da4621-7b38-450f-ab04-2e89039b46a2\" (UID: \"18da4621-7b38-450f-ab04-2e89039b46a2\") " Nov 01 15:40:33.397527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.397500 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume" pod "18da4621-7b38-450f-ab04-2e89039b46a2" (UID: "18da4621-7b38-450f-ab04-2e89039b46a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:33.397690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.397676 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18da4621-7b38-450f-ab04-2e89039b46a2-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:33.397749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.397692 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9v2p\" (UniqueName: \"kubernetes.io/projected/6f0defb1-457c-44e2-a7c3-13c9bc08a042-kube-api-access-c9v2p\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:33.399548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.399525 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp" pod "18da4621-7b38-450f-ab04-2e89039b46a2" (UID: "18da4621-7b38-450f-ab04-2e89039b46a2"). InnerVolumeSpecName "kube-api-access-ktmrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:33.498971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.498891 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktmrp\" (UniqueName: \"kubernetes.io/projected/18da4621-7b38-450f-ab04-2e89039b46a2-kube-api-access-ktmrp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:33.720859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.720822 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz"] Nov 01 15:40:33.724225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.724188 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-1f8e2c-5689c7cb85-rqhhz"] Nov 01 15:40:33.837658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.837615 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:33.837819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:33.837682 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:34.308757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:34.308722 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" Nov 01 15:40:34.309352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:34.308753 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb" event={"ID":"18da4621-7b38-450f-ab04-2e89039b46a2","Type":"ContainerDied","Data":"2eb4fa4a196bc88c331943955e8530a8f17311b7254917a87dd11ca86480bd0c"} Nov 01 15:40:34.309352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:34.308828 5473 scope.go:122] "RemoveContainer" containerID="01130abfafcc34fb2467d26467d26395d841786c19de8491283563f5ce768259" Nov 01 15:40:34.474998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:34.474968 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb"] Nov 01 15:40:34.478188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:34.478176 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-1f8e2c-65d54d57f5-5qwgb"] Nov 01 15:40:35.081859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:35.081823 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" path="/var/lib/kubelet/pods/18da4621-7b38-450f-ab04-2e89039b46a2/volumes" Nov 01 15:40:35.082298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:35.082284 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" path="/var/lib/kubelet/pods/6f0defb1-457c-44e2-a7c3-13c9bc08a042/volumes" Nov 01 15:40:36.086088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:36.086058 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:36.239400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:36.239361 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:36.397609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:36.397529 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:36.578660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:36.578629 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:37.346901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:37.346866 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:37.783052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:37.783023 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b"] Nov 01 15:40:37.783250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:37.783224 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" containerID="cri-o://b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9" gracePeriod=30 Nov 01 15:40:37.789546 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:37.789511 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:38.314065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:38.314023 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt"] Nov 01 15:40:38.414500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:38.414466 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp"] Nov 01 15:40:38.673748 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:38.673655 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:38.838744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:38.838704 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:40.522705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:40.522664 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28"] Nov 01 15:40:40.628648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:40.628606 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9"] Nov 01 15:40:40.919744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:40.919637 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:41.122638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.122606 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:41.320619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.320585 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:41.655272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.655155 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678"] Nov 01 15:40:41.656004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.655981 5473 cpu_manager.go:414] "containerMap: removing container" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" Nov 01 15:40:41.656116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656007 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" Nov 01 15:40:41.656116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656055 5473 cpu_manager.go:414] "containerMap: removing container" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" containerName="payload-pre-processing" Nov 01 15:40:41.656116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656064 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" containerName="payload-pre-processing" Nov 01 15:40:41.656116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656080 5473 cpu_manager.go:414] "containerMap: removing container" podUID="486e25f6-eabc-4114-8d18-5ab19e6868f8" containerName="revoke-keys" Nov 01 15:40:41.656116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656086 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486e25f6-eabc-4114-8d18-5ab19e6868f8" containerName="revoke-keys" Nov 01 15:40:41.656116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656116 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7e9ccfa6-26bc-42e7-82da-7a599002ddd2" containerName="main" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656123 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9ccfa6-26bc-42e7-82da-7a599002ddd2" containerName="main" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656151 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerName="payload-processing" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656159 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerName="payload-processing" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656318 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656336 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="18da4621-7b38-450f-ab04-2e89039b46a2" containerName="payload-pre-processing" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656348 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="486e25f6-eabc-4114-8d18-5ab19e6868f8" containerName="revoke-keys" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656361 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9ccfa6-26bc-42e7-82da-7a599002ddd2" containerName="main" Nov 01 15:40:41.656481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.656374 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0defb1-457c-44e2-a7c3-13c9bc08a042" containerName="payload-processing" Nov 01 15:40:41.661572 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.661553 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.716071 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.716038 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678"] Nov 01 15:40:41.723697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.723667 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:40:41.723843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.723712 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:40:41.800075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.800039 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmjf\" (UniqueName: \"kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.800239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.800189 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.800366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.800350 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.902197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.902158 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmjf\" (UniqueName: \"kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.902407 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.902332 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.902407 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.902366 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.903353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.903329 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.906068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.905995 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.961362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.961336 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmjf\" (UniqueName: \"kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf\") pod \"maas-api-revoke-sub-44aeabe8-ld678\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.975340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.975310 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:41.986629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:41.984064 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 15:40:42.051042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.051016 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 15:40:42.104636 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.104601 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume\") pod \"e598863a-d16a-4959-9a78-3e73ed28ee5a\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " Nov 01 15:40:42.104834 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.104681 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk\" (UniqueName: \"kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk\") pod \"e598863a-d16a-4959-9a78-3e73ed28ee5a\" (UID: \"e598863a-d16a-4959-9a78-3e73ed28ee5a\") " Nov 01 15:40:42.105584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.105556 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume" pod "e598863a-d16a-4959-9a78-3e73ed28ee5a" (UID: "e598863a-d16a-4959-9a78-3e73ed28ee5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:42.117119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.116806 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk" pod "e598863a-d16a-4959-9a78-3e73ed28ee5a" (UID: "e598863a-d16a-4959-9a78-3e73ed28ee5a"). InnerVolumeSpecName "kube-api-access-x8hqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:42.206589 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.206508 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume\") pod \"6047ed06-dc73-49f1-8897-129f144c0afb\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " Nov 01 15:40:42.206786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.206608 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8\" (UniqueName: \"kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8\") pod \"6047ed06-dc73-49f1-8897-129f144c0afb\" (UID: \"6047ed06-dc73-49f1-8897-129f144c0afb\") " Nov 01 15:40:42.206900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.206876 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume" pod "6047ed06-dc73-49f1-8897-129f144c0afb" (UID: "6047ed06-dc73-49f1-8897-129f144c0afb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:42.206978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.206943 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e598863a-d16a-4959-9a78-3e73ed28ee5a-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:42.206978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.206959 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8hqk\" (UniqueName: \"kubernetes.io/projected/e598863a-d16a-4959-9a78-3e73ed28ee5a-kube-api-access-x8hqk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:42.208837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.208813 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8" pod "6047ed06-dc73-49f1-8897-129f144c0afb" (UID: "6047ed06-dc73-49f1-8897-129f144c0afb"). InnerVolumeSpecName "kube-api-access-88hs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:42.308614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.308579 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6047ed06-dc73-49f1-8897-129f144c0afb-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:42.308614 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.308610 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hs8\" (UniqueName: \"kubernetes.io/projected/6047ed06-dc73-49f1-8897-129f144c0afb-kube-api-access-88hs8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:42.368376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.368349 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678"] Nov 01 15:40:42.368502 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:40:42.368409 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ab152a_f149_4057_be2d_f6ba2d9a9b45.slice/crio-754595ef9ce850e3ec36bf28147d43408a2fa7d787300ef46bfa4619097a84c4 WatchSource:0}: Error finding container 754595ef9ce850e3ec36bf28147d43408a2fa7d787300ef46bfa4619097a84c4: Status 404 returned error can't find the container with id 754595ef9ce850e3ec36bf28147d43408a2fa7d787300ef46bfa4619097a84c4 Nov 01 15:40:42.379583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.379553 5473 generic.go:356] "Generic (PLEG): container finished" podID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerID="ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4" exitCode=0 Nov 01 15:40:42.379687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.379599 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" Nov 01 15:40:42.379687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.379632 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" event={"ID":"e598863a-d16a-4959-9a78-3e73ed28ee5a","Type":"ContainerDied","Data":"ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4"} Nov 01 15:40:42.379843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.379687 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq" event={"ID":"e598863a-d16a-4959-9a78-3e73ed28ee5a","Type":"ContainerDied","Data":"24209f5799f6f0042030f624870cc8c1e1560767f6a10a9caf7e656c88b54d2d"} Nov 01 15:40:42.379843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.379720 5473 scope.go:122] "RemoveContainer" containerID="ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4" Nov 01 15:40:42.381304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.381280 5473 generic.go:356] "Generic (PLEG): container finished" podID="6047ed06-dc73-49f1-8897-129f144c0afb" containerID="3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f" exitCode=0 Nov 01 15:40:42.381417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.381332 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" Nov 01 15:40:42.381417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.381386 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" event={"ID":"6047ed06-dc73-49f1-8897-129f144c0afb","Type":"ContainerDied","Data":"3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f"} Nov 01 15:40:42.381529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.381452 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2" event={"ID":"6047ed06-dc73-49f1-8897-129f144c0afb","Type":"ContainerDied","Data":"7024441c6ee57e33ec1650d174648d9ba3e715c888a302dd710fff9cad57a975"} Nov 01 15:40:42.383049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.383012 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" event={"ID":"c8ab152a-f149-4057-be2d-f6ba2d9a9b45","Type":"ContainerStarted","Data":"754595ef9ce850e3ec36bf28147d43408a2fa7d787300ef46bfa4619097a84c4"} Nov 01 15:40:42.395295 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.395274 5473 scope.go:122] "RemoveContainer" containerID="ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4" Nov 01 15:40:42.395649 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:42.395618 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4\": container with ID starting with ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4 not found: ID does not exist" containerID="ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4" Nov 01 15:40:42.395728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.395662 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4"} err="failed to get container status \"ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4\": rpc error: code = NotFound desc = could not find container \"ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4\": container with ID starting with ad0478a71d7fc2fda2f92a2cee74b8700557c25a556d76da46c12c8c11359ac4 not found: ID does not exist" Nov 01 15:40:42.395728 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.395684 5473 scope.go:122] "RemoveContainer" containerID="3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f" Nov 01 15:40:42.406469 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.406450 5473 scope.go:122] "RemoveContainer" containerID="3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f" Nov 01 15:40:42.406726 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:42.406705 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f\": container with ID starting with 3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f not found: ID does not exist" containerID="3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f" Nov 01 15:40:42.406792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.406734 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f"} err="failed to get container status \"3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f\": rpc error: code = NotFound desc = could not find container \"3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f\": container with ID starting with 3a0548ec7107aae03a1c3cb4697cc7ca49240732707373dedfe87a4178c69c5f not found: ID does not exist" Nov 01 15:40:42.545498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.545453 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq"] Nov 01 15:40:42.549031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.549007 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4e07cb-5cc4b45d79-ft6rq"] Nov 01 15:40:42.658179 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.658149 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2"] Nov 01 15:40:42.661770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.661744 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4e07cb-7df5cfddb7-6dbq2"] Nov 01 15:40:42.740106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.740067 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p"] Nov 01 15:40:42.788606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.788576 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:42.842366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.842331 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6"] Nov 01 15:40:42.997054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:42.997018 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:43.089331 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.089293 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" path="/var/lib/kubelet/pods/6047ed06-dc73-49f1-8897-129f144c0afb/volumes" Nov 01 15:40:43.089958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.089931 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" path="/var/lib/kubelet/pods/e598863a-d16a-4959-9a78-3e73ed28ee5a/volumes" Nov 01 15:40:43.113524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.113495 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148203 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148340 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148376 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148407 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148450 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148491 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148533 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148557 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.148593 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket\") pod \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\" (UID: \"d6d2aaa9-1674-45ac-8a9d-c02a4649535f\") " Nov 01 15:40:43.149835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.149608 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:43.149835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.149673 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:43.149969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.149841 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:43.150030 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.150009 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:43.150319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.150285 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:43.157574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.157548 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:40:43.157857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.157649 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "kube-api-access-7p687". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:43.157988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.157814 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:43.158652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.158623 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo" pod "d6d2aaa9-1674-45ac-8a9d-c02a4649535f" (UID: "d6d2aaa9-1674-45ac-8a9d-c02a4649535f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:40:43.175824 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.175791 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:43.250086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250051 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250089 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250108 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250123 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p687\" (UniqueName: \"kubernetes.io/projected/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-kube-api-access-7p687\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250139 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250152 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250165 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250180 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.250274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.250194 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d2aaa9-1674-45ac-8a9d-c02a4649535f-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:43.314789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.314747 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerName="payload-processing" containerID="cri-o://89953ae20769c9b9b3f8acea9cde5703730aca434acada17c6b9d395fa2e8e5a" gracePeriod=25 Nov 01 15:40:43.341035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.341005 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:43.394153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.394121 5473 generic.go:356] "Generic (PLEG): container finished" podID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerID="b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9" exitCode=0 Nov 01 15:40:43.394387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.394170 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" Nov 01 15:40:43.394387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.394204 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" event={"ID":"d6d2aaa9-1674-45ac-8a9d-c02a4649535f","Type":"ContainerDied","Data":"b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9"} Nov 01 15:40:43.394387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.394280 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b" event={"ID":"d6d2aaa9-1674-45ac-8a9d-c02a4649535f","Type":"ContainerDied","Data":"a6d3252eba034a318b224f75c4a76a1f517311e10cbf666f481d2c957d8db5af"} Nov 01 15:40:43.394387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.394308 5473 scope.go:122] "RemoveContainer" containerID="b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9" Nov 01 15:40:43.398599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.398573 5473 generic.go:356] "Generic (PLEG): container finished" podID="c8ab152a-f149-4057-be2d-f6ba2d9a9b45" containerID="99758dde9be5655cce225276845226870becefb1f30932f4dd6181513f0a0f12" exitCode=0 Nov 01 15:40:43.398733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.398616 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" event={"ID":"c8ab152a-f149-4057-be2d-f6ba2d9a9b45","Type":"ContainerDied","Data":"99758dde9be5655cce225276845226870becefb1f30932f4dd6181513f0a0f12"} Nov 01 15:40:43.406252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.406208 5473 scope.go:122] "RemoveContainer" containerID="b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9" Nov 01 15:40:43.406520 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:43.406501 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9\": container with ID starting with b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9 not found: ID does not exist" containerID="b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9" Nov 01 15:40:43.406588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.406533 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9"} err="failed to get container status \"b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9\": rpc error: code = NotFound desc = could not find container \"b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9\": container with ID starting with b562bfc128734390bd19d5d1439dbb1cdff00b93c191dc4b9e6d8c0c4d7078f9 not found: ID does not exist" Nov 01 15:40:43.415647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.415620 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerName="payload-pre-processing" containerID="cri-o://e8bf75a76b4d195dec5b097facd39101a67762095ba08e4f40935ddc8fd5bff4" gracePeriod=25 Nov 01 15:40:43.673387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.673297 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b"] Nov 01 15:40:43.679298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:43.679272 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1f8e2c-openshift-default-7b99b4b599-q2h8b"] Nov 01 15:40:44.593357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.593324 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:44.665924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.665830 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca\") pod \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " Nov 01 15:40:44.665924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.665899 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca\") pod \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " Nov 01 15:40:44.666138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.665928 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf\" (UniqueName: \"kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf\") pod \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\" (UID: \"c8ab152a-f149-4057-be2d-f6ba2d9a9b45\") " Nov 01 15:40:44.666202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.666181 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca" pod "c8ab152a-f149-4057-be2d-f6ba2d9a9b45" (UID: "c8ab152a-f149-4057-be2d-f6ba2d9a9b45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:44.666314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.666298 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:44.668116 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.668081 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca" pod "c8ab152a-f149-4057-be2d-f6ba2d9a9b45" (UID: "c8ab152a-f149-4057-be2d-f6ba2d9a9b45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:40:44.668282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.668190 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf" pod "c8ab152a-f149-4057-be2d-f6ba2d9a9b45" (UID: "c8ab152a-f149-4057-be2d-f6ba2d9a9b45"). InnerVolumeSpecName "kube-api-access-kfmjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:44.767453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.767418 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmjf\" (UniqueName: \"kubernetes.io/projected/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-kube-api-access-kfmjf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:44.767453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:44.767450 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8ab152a-f149-4057-be2d-f6ba2d9a9b45-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:45.084380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.084340 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" path="/var/lib/kubelet/pods/d6d2aaa9-1674-45ac-8a9d-c02a4649535f/volumes" Nov 01 15:40:45.293970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.293929 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:45.424107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.424024 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" Nov 01 15:40:45.424107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.424045 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678" event={"ID":"c8ab152a-f149-4057-be2d-f6ba2d9a9b45","Type":"ContainerDied","Data":"754595ef9ce850e3ec36bf28147d43408a2fa7d787300ef46bfa4619097a84c4"} Nov 01 15:40:45.424107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.424096 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754595ef9ce850e3ec36bf28147d43408a2fa7d787300ef46bfa4619097a84c4" Nov 01 15:40:45.523848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.523795 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" podUID="e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" containerName="payload-processing" containerID="cri-o://87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688" gracePeriod=25 Nov 01 15:40:45.629650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.629608 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" podUID="0d1bef25-851f-4c0a-86eb-bbc95f60d528" containerName="payload-pre-processing" containerID="cri-o://d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6" gracePeriod=25 Nov 01 15:40:45.802863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.802837 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:40:45.885124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.885099 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k\" (UniqueName: \"kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k\") pod \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " Nov 01 15:40:45.885351 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.885336 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume\") pod \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\" (UID: \"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb\") " Nov 01 15:40:45.885828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.885802 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume" pod "e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" (UID: "e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:45.887686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.887660 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k" pod "e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" (UID: "e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb"). InnerVolumeSpecName "kube-api-access-sz97k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:45.897640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.897620 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:40:45.986160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.986075 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km\" (UniqueName: \"kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km\") pod \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " Nov 01 15:40:45.986324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.986304 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume\") pod \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\" (UID: \"0d1bef25-851f-4c0a-86eb-bbc95f60d528\") " Nov 01 15:40:45.986751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.986722 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume" pod "0d1bef25-851f-4c0a-86eb-bbc95f60d528" (UID: "0d1bef25-851f-4c0a-86eb-bbc95f60d528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:40:45.986874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.986749 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:45.986874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.986769 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz97k\" (UniqueName: \"kubernetes.io/projected/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb-kube-api-access-sz97k\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:45.988515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:45.988493 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km" pod "0d1bef25-851f-4c0a-86eb-bbc95f60d528" (UID: "0d1bef25-851f-4c0a-86eb-bbc95f60d528"). InnerVolumeSpecName "kube-api-access-9f8km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:40:46.038302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.038173 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:46.088387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.088356 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1bef25-851f-4c0a-86eb-bbc95f60d528-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:46.088387 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.088384 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f8km\" (UniqueName: \"kubernetes.io/projected/0d1bef25-851f-4c0a-86eb-bbc95f60d528-kube-api-access-9f8km\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:40:46.432117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.432080 5473 generic.go:356] "Generic (PLEG): container finished" podID="0d1bef25-851f-4c0a-86eb-bbc95f60d528" containerID="d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6" exitCode=0 Nov 01 15:40:46.432333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.432135 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" Nov 01 15:40:46.432333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.432165 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" event={"ID":"0d1bef25-851f-4c0a-86eb-bbc95f60d528","Type":"ContainerDied","Data":"d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6"} Nov 01 15:40:46.432333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.432232 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9" event={"ID":"0d1bef25-851f-4c0a-86eb-bbc95f60d528","Type":"ContainerDied","Data":"871e7db5ed5b023a470d8cedbacf69685995f724543e60bda746abd7e4d2699c"} Nov 01 15:40:46.432333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.432256 5473 scope.go:122] "RemoveContainer" containerID="d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6" Nov 01 15:40:46.439152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.439126 5473 generic.go:356] "Generic (PLEG): container finished" podID="e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" containerID="87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688" exitCode=0 Nov 01 15:40:46.439312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.439170 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" Nov 01 15:40:46.439312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.439209 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" event={"ID":"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb","Type":"ContainerDied","Data":"87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688"} Nov 01 15:40:46.439312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.439281 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28" event={"ID":"e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb","Type":"ContainerDied","Data":"115817ac6f5bbb9b984c040c0fda30e856df389e49f9f0e1ac79e43ed0dace24"} Nov 01 15:40:46.443037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.443018 5473 scope.go:122] "RemoveContainer" containerID="d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6" Nov 01 15:40:46.443301 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:46.443282 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6\": container with ID starting with d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6 not found: ID does not exist" containerID="d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6" Nov 01 15:40:46.443393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.443308 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6"} err="failed to get container status \"d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6\": rpc error: code = NotFound desc = could not find container \"d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6\": container with ID starting with d586b24605d83b846517d750e2e82d4bca67d1a1c0fbf164dd1012c48f0843f6 not found: ID does not exist" Nov 01 15:40:46.443393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.443325 5473 scope.go:122] "RemoveContainer" containerID="87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688" Nov 01 15:40:46.454251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.454233 5473 scope.go:122] "RemoveContainer" containerID="87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688" Nov 01 15:40:46.454533 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:40:46.454514 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688\": container with ID starting with 87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688 not found: ID does not exist" containerID="87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688" Nov 01 15:40:46.454588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.454541 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688"} err="failed to get container status \"87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688\": rpc error: code = NotFound desc = could not find container \"87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688\": container with ID starting with 87dd849893ff39bf8fc0dedf60accac1c01f6311a497bd8e0b06f5d004774688 not found: ID does not exist" Nov 01 15:40:46.596886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.596853 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28"] Nov 01 15:40:46.646989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.646954 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-55943a-5c49566989-x8g28"] Nov 01 15:40:46.756632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.756549 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9"] Nov 01 15:40:46.759843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:46.759820 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-55943a-547fd598c-k74g9"] Nov 01 15:40:47.084049 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:47.084008 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1bef25-851f-4c0a-86eb-bbc95f60d528" path="/var/lib/kubelet/pods/0d1bef25-851f-4c0a-86eb-bbc95f60d528/volumes" Nov 01 15:40:47.084462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:47.084451 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" path="/var/lib/kubelet/pods/e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb/volumes" Nov 01 15:40:47.741283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:47.741241 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerName="payload-processing" containerID="cri-o://649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d" gracePeriod=25 Nov 01 15:40:47.842804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:47.842762 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerName="payload-pre-processing" containerID="cri-o://38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19" gracePeriod=25 Nov 01 15:40:48.997185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:48.997149 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:48.997185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:48.997175 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:49.366624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:49.366578 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:50.764917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:50.764856 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:51.082839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:51.082740 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:51.082839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:51.082752 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:52.399766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:52.399730 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:55.081611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:55.081563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:55.428236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:55.428116 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:58.901696 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:58.901664 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:58.996871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:58.996836 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:58.997073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:58.996951 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:40:59.031750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:59.031713 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8"] Nov 01 15:40:59.107361 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:59.107328 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:40:59.966107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:40:59.966061 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:00.292362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:00.292328 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:00.497164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:00.497123 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:01.082270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:01.082233 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:01.082270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:01.082242 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:01.108419 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:01.108381 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:41:01.500032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:01.499948 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:01.664140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:01.664108 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:02.838958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:02.838860 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg"] Nov 01 15:41:02.945079 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:02.945045 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq"] Nov 01 15:41:03.198934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.198851 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:03.350264 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.350207 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:03.562432 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.562403 5473 generic.go:356] "Generic (PLEG): container finished" podID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerID="e8bf75a76b4d195dec5b097facd39101a67762095ba08e4f40935ddc8fd5bff4" exitCode=0 Nov 01 15:41:03.562583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.562465 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" event={"ID":"d800b5c5-160b-4c46-96ae-4956d1777d09","Type":"ContainerDied","Data":"e8bf75a76b4d195dec5b097facd39101a67762095ba08e4f40935ddc8fd5bff4"} Nov 01 15:41:03.564289 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.564264 5473 generic.go:356] "Generic (PLEG): container finished" podID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerID="89953ae20769c9b9b3f8acea9cde5703730aca434acada17c6b9d395fa2e8e5a" exitCode=0 Nov 01 15:41:03.564424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.564304 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" event={"ID":"252062aa-197d-4f99-b9e1-2c656bfe26bb","Type":"ContainerDied","Data":"89953ae20769c9b9b3f8acea9cde5703730aca434acada17c6b9d395fa2e8e5a"} Nov 01 15:41:03.564424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.564341 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" event={"ID":"252062aa-197d-4f99-b9e1-2c656bfe26bb","Type":"ContainerDied","Data":"259728db4b58c8e9f0e3e062650f977ec25c377942f82c02fec20f1f18313cae"} Nov 01 15:41:03.564424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.564367 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259728db4b58c8e9f0e3e062650f977ec25c377942f82c02fec20f1f18313cae" Nov 01 15:41:03.579472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.579445 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:41:03.681496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.681469 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:41:03.704091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.704061 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r\" (UniqueName: \"kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r\") pod \"252062aa-197d-4f99-b9e1-2c656bfe26bb\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " Nov 01 15:41:03.704307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.704164 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume\") pod \"252062aa-197d-4f99-b9e1-2c656bfe26bb\" (UID: \"252062aa-197d-4f99-b9e1-2c656bfe26bb\") " Nov 01 15:41:03.704668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.704640 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume" pod "252062aa-197d-4f99-b9e1-2c656bfe26bb" (UID: "252062aa-197d-4f99-b9e1-2c656bfe26bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:03.706765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.706742 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r" pod "252062aa-197d-4f99-b9e1-2c656bfe26bb" (UID: "252062aa-197d-4f99-b9e1-2c656bfe26bb"). InnerVolumeSpecName "kube-api-access-zpp8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:03.805900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.805861 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume\") pod \"d800b5c5-160b-4c46-96ae-4956d1777d09\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " Nov 01 15:41:03.806083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.805948 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws\" (UniqueName: \"kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws\") pod \"d800b5c5-160b-4c46-96ae-4956d1777d09\" (UID: \"d800b5c5-160b-4c46-96ae-4956d1777d09\") " Nov 01 15:41:03.806327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.806306 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume" pod "d800b5c5-160b-4c46-96ae-4956d1777d09" (UID: "d800b5c5-160b-4c46-96ae-4956d1777d09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:03.806380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.806366 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpp8r\" (UniqueName: \"kubernetes.io/projected/252062aa-197d-4f99-b9e1-2c656bfe26bb-kube-api-access-zpp8r\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:03.806418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.806382 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252062aa-197d-4f99-b9e1-2c656bfe26bb-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:03.808238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.808201 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws" pod "d800b5c5-160b-4c46-96ae-4956d1777d09" (UID: "d800b5c5-160b-4c46-96ae-4956d1777d09"). InnerVolumeSpecName "kube-api-access-k28ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:03.907181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.907148 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d800b5c5-160b-4c46-96ae-4956d1777d09-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:03.907181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:03.907174 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28ws\" (UniqueName: \"kubernetes.io/projected/d800b5c5-160b-4c46-96ae-4956d1777d09-kube-api-access-k28ws\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:04.241535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.241445 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:04.404129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.404079 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:04.573038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.573005 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" Nov 01 15:41:04.573038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.573025 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp" event={"ID":"d800b5c5-160b-4c46-96ae-4956d1777d09","Type":"ContainerDied","Data":"d86b5dc95001aa054dbb6da1b619d72468381cb8ce55599a6616105724c93a9a"} Nov 01 15:41:04.573326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.573089 5473 scope.go:122] "RemoveContainer" containerID="e8bf75a76b4d195dec5b097facd39101a67762095ba08e4f40935ddc8fd5bff4" Nov 01 15:41:04.573326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.573014 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt" Nov 01 15:41:04.740296 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.740264 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp"] Nov 01 15:41:04.791189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.791158 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-ca0f9e-57f84c54c-z5dvp"] Nov 01 15:41:04.899420 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.899328 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt"] Nov 01 15:41:04.902857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:04.902830 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ca0f9e-76484b565f-rnqgt"] Nov 01 15:41:05.082131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:05.082097 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" path="/var/lib/kubelet/pods/252062aa-197d-4f99-b9e1-2c656bfe26bb/volumes" Nov 01 15:41:05.082553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:05.082543 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" path="/var/lib/kubelet/pods/d800b5c5-160b-4c46-96ae-4956d1777d09/volumes" Nov 01 15:41:05.774149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:05.774118 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:05.961691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:05.961657 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:06.130532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:06.130440 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:06.518820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:06.518790 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:07.367312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:07.367272 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:07.742663 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:07.742587 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp"] Nov 01 15:41:07.742813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:07.742785 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" containerID="cri-o://bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624" gracePeriod=30 Nov 01 15:41:07.774323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:07.774296 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:07.839742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:07.839678 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerName="payload-processing" containerID="cri-o://ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f" gracePeriod=25 Nov 01 15:41:07.946270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:07.946209 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" podUID="94500465-cb49-41e5-9378-71b86396fda9" containerName="payload-pre-processing" containerID="cri-o://de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93" gracePeriod=25 Nov 01 15:41:08.014009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.013984 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 15:41:08.058831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.058796 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume\") pod \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " Nov 01 15:41:08.059009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.058880 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8\" (UniqueName: \"kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8\") pod \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\" (UID: \"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7\") " Nov 01 15:41:08.059180 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.059156 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume" pod "6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" (UID: "6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:08.059446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.059427 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:08.061382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.061351 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8" pod "6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" (UID: "6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7"). InnerVolumeSpecName "kube-api-access-r48d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:08.080833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.080806 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 15:41:08.159943 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.159904 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume\") pod \"a17c119a-7a1b-48d4-b093-99b268dc1569\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " Nov 01 15:41:08.160152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.159998 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4\" (UniqueName: \"kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4\") pod \"a17c119a-7a1b-48d4-b093-99b268dc1569\" (UID: \"a17c119a-7a1b-48d4-b093-99b268dc1569\") " Nov 01 15:41:08.160370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.160334 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume" pod "a17c119a-7a1b-48d4-b093-99b268dc1569" (UID: "a17c119a-7a1b-48d4-b093-99b268dc1569"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:08.160539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.160519 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a17c119a-7a1b-48d4-b093-99b268dc1569-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:08.160628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.160545 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r48d8\" (UniqueName: \"kubernetes.io/projected/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7-kube-api-access-r48d8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:08.162696 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.162668 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4" pod "a17c119a-7a1b-48d4-b093-99b268dc1569" (UID: "a17c119a-7a1b-48d4-b093-99b268dc1569"). InnerVolumeSpecName "kube-api-access-xmtv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:08.246762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.246673 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:08.261197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.261161 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmtv4\" (UniqueName: \"kubernetes.io/projected/a17c119a-7a1b-48d4-b093-99b268dc1569-kube-api-access-xmtv4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:08.415402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.415363 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:08.580327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.580292 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:08.601903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.601867 5473 generic.go:356] "Generic (PLEG): container finished" podID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerID="649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d" exitCode=0 Nov 01 15:41:08.602048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.601918 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" Nov 01 15:41:08.602048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.601950 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" event={"ID":"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7","Type":"ContainerDied","Data":"649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d"} Nov 01 15:41:08.602048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.602007 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p" event={"ID":"6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7","Type":"ContainerDied","Data":"e1acfc6b56b10eaa2abbed5b720a0b7b0fb45c904885510cde465576d16fd388"} Nov 01 15:41:08.602048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.602032 5473 scope.go:122] "RemoveContainer" containerID="649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d" Nov 01 15:41:08.603660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.603634 5473 generic.go:356] "Generic (PLEG): container finished" podID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerID="38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19" exitCode=0 Nov 01 15:41:08.603764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.603669 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" Nov 01 15:41:08.603764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.603672 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" event={"ID":"a17c119a-7a1b-48d4-b093-99b268dc1569","Type":"ContainerDied","Data":"38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19"} Nov 01 15:41:08.603764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.603711 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6" event={"ID":"a17c119a-7a1b-48d4-b093-99b268dc1569","Type":"ContainerDied","Data":"d3124103c2608ca7dc2ff12101d06f93a0a6fee720f53f132e22bdcb2d272cf7"} Nov 01 15:41:08.612937 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.612915 5473 scope.go:122] "RemoveContainer" containerID="649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d" Nov 01 15:41:08.613252 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:08.613228 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d\": container with ID starting with 649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d not found: ID does not exist" containerID="649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d" Nov 01 15:41:08.613340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.613263 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d"} err="failed to get container status \"649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d\": rpc error: code = NotFound desc = could not find container \"649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d\": container with ID starting with 649402696f40c67a1555d97ac5f85e9ef49be3ccd912f845ed925e3668c1879d not found: ID does not exist" Nov 01 15:41:08.613340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.613286 5473 scope.go:122] "RemoveContainer" containerID="38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19" Nov 01 15:41:08.622605 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.622586 5473 scope.go:122] "RemoveContainer" containerID="38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19" Nov 01 15:41:08.622881 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:08.622863 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19\": container with ID starting with 38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19 not found: ID does not exist" containerID="38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19" Nov 01 15:41:08.622941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.622888 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19"} err="failed to get container status \"38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19\": rpc error: code = NotFound desc = could not find container \"38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19\": container with ID starting with 38864ccba036a1ecdfbd3e66acc079ef3e108835e94fe416440c612bb71dec19 not found: ID does not exist" Nov 01 15:41:08.769034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769000 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq"] Nov 01 15:41:08.769618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769600 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0d1bef25-851f-4c0a-86eb-bbc95f60d528" containerName="payload-pre-processing" Nov 01 15:41:08.769669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769620 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1bef25-851f-4c0a-86eb-bbc95f60d528" containerName="payload-pre-processing" Nov 01 15:41:08.769669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769643 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerName="payload-processing" Nov 01 15:41:08.769669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769649 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerName="payload-processing" Nov 01 15:41:08.769669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769662 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerName="payload-pre-processing" Nov 01 15:41:08.769669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769667 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerName="payload-pre-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769674 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c8ab152a-f149-4057-be2d-f6ba2d9a9b45" containerName="revoke-keys" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769680 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ab152a-f149-4057-be2d-f6ba2d9a9b45" containerName="revoke-keys" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769691 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" containerName="payload-pre-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769695 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" containerName="payload-pre-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769705 5473 cpu_manager.go:414] "containerMap: removing container" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769709 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769720 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769724 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769732 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerName="payload-pre-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769737 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerName="payload-pre-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769750 5473 cpu_manager.go:414] "containerMap: removing container" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769755 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769762 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769767 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769776 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769780 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769856 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d800b5c5-160b-4c46-96ae-4956d1777d09" containerName="payload-pre-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769866 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="252062aa-197d-4f99-b9e1-2c656bfe26bb" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769872 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a9e3ac-dcad-4ec3-8e8d-e7956314f3fb" containerName="payload-processing" Nov 01 15:41:08.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769879 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" containerName="payload-processing" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769887 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1bef25-851f-4c0a-86eb-bbc95f60d528" containerName="payload-pre-processing" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769895 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e598863a-d16a-4959-9a78-3e73ed28ee5a" containerName="payload-processing" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769902 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" containerName="payload-pre-processing" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769906 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ab152a-f149-4057-be2d-f6ba2d9a9b45" containerName="revoke-keys" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769912 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6047ed06-dc73-49f1-8897-129f144c0afb" containerName="payload-pre-processing" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769919 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c16441-be77-4a41-b90d-77b04df67172" containerName="istio-proxy" Nov 01 15:41:08.770502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.769924 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d2aaa9-1674-45ac-8a9d-c02a4649535f" containerName="istio-proxy" Nov 01 15:41:08.774765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.774749 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p"] Nov 01 15:41:08.774840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.774829 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.818237 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.818182 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-536a87-96486c8f5-gsf7p"] Nov 01 15:41:08.827118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.827074 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:41:08.827416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.827393 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:41:08.828706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.828681 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq"] Nov 01 15:41:08.867603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.867521 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.867742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.867680 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpzt\" (UniqueName: \"kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.867742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.867733 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.878613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.878579 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6"] Nov 01 15:41:08.881554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.881530 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-536a87-55fd4f65f5-x54m6"] Nov 01 15:41:08.968644 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.968607 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.968853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.968698 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.968853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.968832 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpzt\" (UniqueName: \"kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.969523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.969497 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:08.971250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:08.971228 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:09.025779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.025747 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpzt\" (UniqueName: \"kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt\") pod \"maas-api-revoke-sub-4c61b32d-pjtrq\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:09.081337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.081302 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7" path="/var/lib/kubelet/pods/6738a0ee-98a9-4c7d-a2d4-4d8ebc7350b7/volumes" Nov 01 15:41:09.081708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.081695 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17c119a-7a1b-48d4-b093-99b268dc1569" path="/var/lib/kubelet/pods/a17c119a-7a1b-48d4-b093-99b268dc1569/volumes" Nov 01 15:41:09.088152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.088130 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:09.468434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.468400 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq"] Nov 01 15:41:09.468828 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:41:09.468802 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29b786c_35d9_4eb5_be98_1dd51f5dac65.slice/crio-29c92d29ea64ea71b68c23e9c7b736d61f8bb15c22bd333da971e3973f942c06 WatchSource:0}: Error finding container 29c92d29ea64ea71b68c23e9c7b736d61f8bb15c22bd333da971e3973f942c06: Status 404 returned error can't find the container with id 29c92d29ea64ea71b68c23e9c7b736d61f8bb15c22bd333da971e3973f942c06 Nov 01 15:41:09.612767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.612730 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" event={"ID":"d29b786c-35d9-4eb5-be98-1dd51f5dac65","Type":"ContainerStarted","Data":"29c92d29ea64ea71b68c23e9c7b736d61f8bb15c22bd333da971e3973f942c06"} Nov 01 15:41:09.637569 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.637543 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:09.966243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.966196 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:09.969011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.968982 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:09.969176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:09.969008 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" podUID="94500465-cb49-41e5-9378-71b86396fda9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:10.502196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:10.502153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:10.622645 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:10.622610 5473 generic.go:356] "Generic (PLEG): container finished" podID="d29b786c-35d9-4eb5-be98-1dd51f5dac65" containerID="779bbdf74079601aac6e51f973aa1fb8efe567ccf11ad481fcb7b7dfc270bf2e" exitCode=0 Nov 01 15:41:10.622830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:10.622652 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" event={"ID":"d29b786c-35d9-4eb5-be98-1dd51f5dac65","Type":"ContainerDied","Data":"779bbdf74079601aac6e51f973aa1fb8efe567ccf11ad481fcb7b7dfc270bf2e"} Nov 01 15:41:11.057397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.057364 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:11.769873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.769843 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:11.799948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.799916 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca\") pod \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " Nov 01 15:41:11.800105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.800068 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt\" (UniqueName: \"kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt\") pod \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " Nov 01 15:41:11.800191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.800128 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca\") pod \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\" (UID: \"d29b786c-35d9-4eb5-be98-1dd51f5dac65\") " Nov 01 15:41:11.800584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.800557 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca" pod "d29b786c-35d9-4eb5-be98-1dd51f5dac65" (UID: "d29b786c-35d9-4eb5-be98-1dd51f5dac65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:11.802154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.802131 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt" pod "d29b786c-35d9-4eb5-be98-1dd51f5dac65" (UID: "d29b786c-35d9-4eb5-be98-1dd51f5dac65"). InnerVolumeSpecName "kube-api-access-mtpzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:11.802242 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.802191 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca" pod "d29b786c-35d9-4eb5-be98-1dd51f5dac65" (UID: "d29b786c-35d9-4eb5-be98-1dd51f5dac65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:41:11.902429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.902353 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:11.902429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.902386 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpzt\" (UniqueName: \"kubernetes.io/projected/d29b786c-35d9-4eb5-be98-1dd51f5dac65-kube-api-access-mtpzt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:11.902429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:11.902397 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29b786c-35d9-4eb5-be98-1dd51f5dac65-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:12.638876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:12.638844 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" Nov 01 15:41:12.639064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:12.638845 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq" event={"ID":"d29b786c-35d9-4eb5-be98-1dd51f5dac65","Type":"ContainerDied","Data":"29c92d29ea64ea71b68c23e9c7b736d61f8bb15c22bd333da971e3973f942c06"} Nov 01 15:41:12.639064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:12.638975 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c92d29ea64ea71b68c23e9c7b736d61f8bb15c22bd333da971e3973f942c06" Nov 01 15:41:13.001958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.001930 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:41:13.115641 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115605 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.115797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115678 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.115797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115699 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.115797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115727 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.115957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115866 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.116024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115953 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.116024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.115986 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.116137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116032 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.116137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116068 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:13.116137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116108 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token\") pod \"46ceda98-4397-4c78-91ed-655e5fe7523e\" (UID: \"46ceda98-4397-4c78-91ed-655e5fe7523e\") " Nov 01 15:41:13.116344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116294 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:13.116344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116298 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:13.116344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116312 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:13.116497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116433 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:13.116653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116630 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.116735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116657 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.116735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116673 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.116735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116688 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ceda98-4397-4c78-91ed-655e5fe7523e-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.116735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.116702 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.118180 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.118158 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "kube-api-access-fn9ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:13.118452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.118430 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:13.123281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.118694 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:13.126355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.126327 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo" pod "46ceda98-4397-4c78-91ed-655e5fe7523e" (UID: "46ceda98-4397-4c78-91ed-655e5fe7523e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:41:13.172764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.172667 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm"] Nov 01 15:41:13.217555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.217521 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn9ws\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-kube-api-access-fn9ws\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.217555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.217552 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.217555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.217562 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.217810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.217570 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ceda98-4397-4c78-91ed-655e5fe7523e-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:13.255980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.255944 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:13.647027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.646983 5473 generic.go:356] "Generic (PLEG): container finished" podID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerID="bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624" exitCode=0 Nov 01 15:41:13.647411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.647064 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" Nov 01 15:41:13.647411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.647067 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" event={"ID":"46ceda98-4397-4c78-91ed-655e5fe7523e","Type":"ContainerDied","Data":"bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624"} Nov 01 15:41:13.647411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.647153 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp" event={"ID":"46ceda98-4397-4c78-91ed-655e5fe7523e","Type":"ContainerDied","Data":"3867a83777a2a99fe44c4824d8ca77691428b1dfe49cfe2441ff590007fb2562"} Nov 01 15:41:13.647411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.647179 5473 scope.go:122] "RemoveContainer" containerID="bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624" Nov 01 15:41:13.658552 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.658529 5473 scope.go:122] "RemoveContainer" containerID="b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55" Nov 01 15:41:13.668200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.668178 5473 scope.go:122] "RemoveContainer" containerID="bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624" Nov 01 15:41:13.668559 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:13.668522 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624\": container with ID starting with bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624 not found: ID does not exist" containerID="bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624" Nov 01 15:41:13.668664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.668567 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624"} err="failed to get container status \"bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624\": rpc error: code = NotFound desc = could not find container \"bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624\": container with ID starting with bb9c8500eab5e7906fc9bae1314f2bbd10331769178ec7c279dcc05e1bcda624 not found: ID does not exist" Nov 01 15:41:13.668664 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.668594 5473 scope.go:122] "RemoveContainer" containerID="b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55" Nov 01 15:41:13.668907 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:13.668884 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55\": container with ID starting with b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55 not found: ID does not exist" containerID="b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55" Nov 01 15:41:13.668979 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.668913 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55"} err="failed to get container status \"b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55\": rpc error: code = NotFound desc = could not find container \"b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55\": container with ID starting with b345764805477f1207a5bcafa38e51d1a4ea65d777298e00fd7c59392781ba55 not found: ID does not exist" Nov 01 15:41:13.813779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.813742 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp"] Nov 01 15:41:13.865085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:13.865038 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ca0f9e-openshift-default-7c6f949d4f-k9cgp"] Nov 01 15:41:14.052192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.052126 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" podUID="ce984260-255c-4072-b151-93e42303e8c0" containerName="main" containerID="cri-o://0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10" gracePeriod=45 Nov 01 15:41:14.124592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.124520 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:14.322859 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.322836 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:41:14.427547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427513 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm\") pod \"ce984260-255c-4072-b151-93e42303e8c0\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " Nov 01 15:41:14.427763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427559 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache\") pod \"ce984260-255c-4072-b151-93e42303e8c0\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " Nov 01 15:41:14.427763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427620 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz\" (UniqueName: \"kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz\") pod \"ce984260-255c-4072-b151-93e42303e8c0\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " Nov 01 15:41:14.427763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427659 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home\") pod \"ce984260-255c-4072-b151-93e42303e8c0\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " Nov 01 15:41:14.427763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427696 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs\") pod \"ce984260-255c-4072-b151-93e42303e8c0\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " Nov 01 15:41:14.427763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427743 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir\") pod \"ce984260-255c-4072-b151-93e42303e8c0\" (UID: \"ce984260-255c-4072-b151-93e42303e8c0\") " Nov 01 15:41:14.428054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427920 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache" pod "ce984260-255c-4072-b151-93e42303e8c0" (UID: "ce984260-255c-4072-b151-93e42303e8c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:14.428054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.427971 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home" pod "ce984260-255c-4072-b151-93e42303e8c0" (UID: "ce984260-255c-4072-b151-93e42303e8c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:14.428172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.428146 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir" pod "ce984260-255c-4072-b151-93e42303e8c0" (UID: "ce984260-255c-4072-b151-93e42303e8c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:14.428314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.428284 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:14.428314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.428308 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:14.428433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.428325 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:14.429958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.429936 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm" pod "ce984260-255c-4072-b151-93e42303e8c0" (UID: "ce984260-255c-4072-b151-93e42303e8c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:14.430050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.430000 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz" pod "ce984260-255c-4072-b151-93e42303e8c0" (UID: "ce984260-255c-4072-b151-93e42303e8c0"). InnerVolumeSpecName "kube-api-access-72rsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:14.430050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.430026 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs" pod "ce984260-255c-4072-b151-93e42303e8c0" (UID: "ce984260-255c-4072-b151-93e42303e8c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:41:14.528965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.528926 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce984260-255c-4072-b151-93e42303e8c0-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:14.528965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.528961 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72rsz\" (UniqueName: \"kubernetes.io/projected/ce984260-255c-4072-b151-93e42303e8c0-kube-api-access-72rsz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:14.528965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.528972 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce984260-255c-4072-b151-93e42303e8c0-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:14.656519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.656426 5473 generic.go:356] "Generic (PLEG): container finished" podID="ce984260-255c-4072-b151-93e42303e8c0" containerID="0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10" exitCode=0 Nov 01 15:41:14.656519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.656473 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" Nov 01 15:41:14.656519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.656481 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" event={"ID":"ce984260-255c-4072-b151-93e42303e8c0","Type":"ContainerDied","Data":"0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10"} Nov 01 15:41:14.656519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.656520 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8" event={"ID":"ce984260-255c-4072-b151-93e42303e8c0","Type":"ContainerDied","Data":"0ba83364e85a7567a0673c43ab4c2271531b2992825e6c6bc635ac6b6c6726f5"} Nov 01 15:41:14.656842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.656541 5473 scope.go:122] "RemoveContainer" containerID="0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10" Nov 01 15:41:14.671476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.671450 5473 scope.go:122] "RemoveContainer" containerID="0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10" Nov 01 15:41:14.671797 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:14.671776 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10\": container with ID starting with 0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10 not found: ID does not exist" containerID="0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10" Nov 01 15:41:14.671892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.671802 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10"} err="failed to get container status \"0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10\": rpc error: code = NotFound desc = could not find container \"0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10\": container with ID starting with 0ba3a8e3f936ad213191c83d0adf93d6195ce94da7cd88e5111f95ddbec9ef10 not found: ID does not exist" Nov 01 15:41:14.821534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.821486 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8"] Nov 01 15:41:14.824515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:14.824490 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3f45c3/test-model-3f45c3-kserve-f7bd7f68c-2pzp8"] Nov 01 15:41:15.020714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:15.020682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:15.081622 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:15.081590 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" path="/var/lib/kubelet/pods/46ceda98-4397-4c78-91ed-655e5fe7523e/volumes" Nov 01 15:41:15.082040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:15.082027 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce984260-255c-4072-b151-93e42303e8c0" path="/var/lib/kubelet/pods/ce984260-255c-4072-b151-93e42303e8c0/volumes" Nov 01 15:41:16.002495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:16.002462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:16.699318 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:16.699282 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:19.359445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:19.359407 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:19.968515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:19.968475 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" podUID="94500465-cb49-41e5-9378-71b86396fda9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:19.968710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:19.968486 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:20.619838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:20.619797 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:22.282514 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:22.282480 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:22.631539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:22.631446 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:22.882010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:22.881925 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:24.106477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.106446 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:24.527293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.527252 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm"] Nov 01 15:41:24.527556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.527522 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" containerID="cri-o://55591be9f016706e984e911b34cc6faa497a1c1ca775b220c9080d4acaeb205a" gracePeriod=30 Nov 01 15:41:24.564363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.564323 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:24.702202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702156 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88"] Nov 01 15:41:24.702805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702784 5473 cpu_manager.go:414] "containerMap: removing container" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702806 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702817 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d29b786c-35d9-4eb5-be98-1dd51f5dac65" containerName="revoke-keys" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702824 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29b786c-35d9-4eb5-be98-1dd51f5dac65" containerName="revoke-keys" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702880 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ce984260-255c-4072-b151-93e42303e8c0" containerName="main" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702889 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce984260-255c-4072-b151-93e42303e8c0" containerName="main" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702902 5473 cpu_manager.go:414] "containerMap: removing container" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" Nov 01 15:41:24.702921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.702908 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" Nov 01 15:41:24.703338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.703029 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" Nov 01 15:41:24.703338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.703042 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29b786c-35d9-4eb5-be98-1dd51f5dac65" containerName="revoke-keys" Nov 01 15:41:24.703338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.703062 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce984260-255c-4072-b151-93e42303e8c0" containerName="main" Nov 01 15:41:24.706568 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.706547 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.763003 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.762967 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88"] Nov 01 15:41:24.814118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.814033 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:41:24.814118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.814053 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:41:24.839667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.839634 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.839667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.839671 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.839936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.839826 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q9dv\" (UniqueName: \"kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.940815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.940782 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q9dv\" (UniqueName: \"kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.941066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.940855 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.941066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.940876 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.941600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.941575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.943403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.943379 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:24.998383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:24.998354 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q9dv\" (UniqueName: \"kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv\") pod \"maas-api-revoke-sub-ab2218b8-c5f88\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:25.020325 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:25.020290 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:25.257597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:25.257566 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:25.394694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:25.394656 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88"] Nov 01 15:41:25.395540 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:41:25.395506 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f633d5b_4ebc_4718_b1ee_2e0f33baba07.slice/crio-12a69b6da22dc842fc54b104c5f3b93deafde07d403bf65879d588f6d5e8ab6e WatchSource:0}: Error finding container 12a69b6da22dc842fc54b104c5f3b93deafde07d403bf65879d588f6d5e8ab6e: Status 404 returned error can't find the container with id 12a69b6da22dc842fc54b104c5f3b93deafde07d403bf65879d588f6d5e8ab6e Nov 01 15:41:25.549480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:25.549448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:25.709854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:25.709819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:25.734839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:25.734812 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" event={"ID":"0f633d5b-4ebc-4718-b1ee-2e0f33baba07","Type":"ContainerStarted","Data":"12a69b6da22dc842fc54b104c5f3b93deafde07d403bf65879d588f6d5e8ab6e"} Nov 01 15:41:26.289866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:26.289837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:26.438906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:26.438867 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv"] Nov 01 15:41:26.508440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:26.508406 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7"] Nov 01 15:41:26.743128 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:26.743039 5473 generic.go:356] "Generic (PLEG): container finished" podID="0f633d5b-4ebc-4718-b1ee-2e0f33baba07" containerID="f251bc2cee24b0c7d6d48dc5072fc3845b6d78cafbb1460be6a32fd9036563c1" exitCode=0 Nov 01 15:41:26.743370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:26.743137 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" event={"ID":"0f633d5b-4ebc-4718-b1ee-2e0f33baba07","Type":"ContainerDied","Data":"f251bc2cee24b0c7d6d48dc5072fc3845b6d78cafbb1460be6a32fd9036563c1"} Nov 01 15:41:26.793140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:26.793112 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:27.050083 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:27.050041 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:27.256106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:27.256068 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:27.333019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:27.332928 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8"] Nov 01 15:41:27.333537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:27.333184 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" containerID="cri-o://dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a" gracePeriod=30 Nov 01 15:41:27.432780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:27.432745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:28.030040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.030010 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:28.100479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.100455 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:41:28.101707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.101684 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:28.177270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177235 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7\" (UniqueName: \"kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7\") pod \"abe101fc-9a38-4bcf-bf63-002bbf93709d\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " Nov 01 15:41:28.177427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177308 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv\" (UniqueName: \"kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv\") pod \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " Nov 01 15:41:28.177427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177338 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca\") pod \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " Nov 01 15:41:28.177427 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177363 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume\") pod \"abe101fc-9a38-4bcf-bf63-002bbf93709d\" (UID: \"abe101fc-9a38-4bcf-bf63-002bbf93709d\") " Nov 01 15:41:28.177643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177476 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca\") pod \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\" (UID: \"0f633d5b-4ebc-4718-b1ee-2e0f33baba07\") " Nov 01 15:41:28.177958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177926 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume" pod "abe101fc-9a38-4bcf-bf63-002bbf93709d" (UID: "abe101fc-9a38-4bcf-bf63-002bbf93709d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:28.177958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.177943 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca" pod "0f633d5b-4ebc-4718-b1ee-2e0f33baba07" (UID: "0f633d5b-4ebc-4718-b1ee-2e0f33baba07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:28.179472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.179451 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv" pod "0f633d5b-4ebc-4718-b1ee-2e0f33baba07" (UID: "0f633d5b-4ebc-4718-b1ee-2e0f33baba07"). InnerVolumeSpecName "kube-api-access-5q9dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:28.179893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.179872 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca" pod "0f633d5b-4ebc-4718-b1ee-2e0f33baba07" (UID: "0f633d5b-4ebc-4718-b1ee-2e0f33baba07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:41:28.179893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.179881 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7" pod "abe101fc-9a38-4bcf-bf63-002bbf93709d" (UID: "abe101fc-9a38-4bcf-bf63-002bbf93709d"). InnerVolumeSpecName "kube-api-access-cnlt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:28.193953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.193926 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" containerID="cri-o://a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a" gracePeriod=45 Nov 01 15:41:28.194964 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.194947 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:41:28.278693 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.278615 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9\" (UniqueName: \"kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9\") pod \"94500465-cb49-41e5-9378-71b86396fda9\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " Nov 01 15:41:28.278847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.278709 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume\") pod \"94500465-cb49-41e5-9378-71b86396fda9\" (UID: \"94500465-cb49-41e5-9378-71b86396fda9\") " Nov 01 15:41:28.279082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279047 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.279130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279087 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnlt7\" (UniqueName: \"kubernetes.io/projected/abe101fc-9a38-4bcf-bf63-002bbf93709d-kube-api-access-cnlt7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.279130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279103 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q9dv\" (UniqueName: \"kubernetes.io/projected/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-kube-api-access-5q9dv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.279130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279119 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f633d5b-4ebc-4718-b1ee-2e0f33baba07-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.279279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279134 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe101fc-9a38-4bcf-bf63-002bbf93709d-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.279279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279137 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume" pod "94500465-cb49-41e5-9378-71b86396fda9" (UID: "94500465-cb49-41e5-9378-71b86396fda9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:28.279449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.279418 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused" Nov 01 15:41:28.281142 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.281119 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9" pod "94500465-cb49-41e5-9378-71b86396fda9" (UID: "94500465-cb49-41e5-9378-71b86396fda9"). InnerVolumeSpecName "kube-api-access-x8fs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:28.381093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.381049 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8fs9\" (UniqueName: \"kubernetes.io/projected/94500465-cb49-41e5-9378-71b86396fda9-kube-api-access-x8fs9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.381093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.381095 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94500465-cb49-41e5-9378-71b86396fda9-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.427111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.427085 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 15:41:28.583867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.583766 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs\") pod \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " Nov 01 15:41:28.583867 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.583828 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache\") pod \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " Nov 01 15:41:28.584147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.583897 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8\" (UniqueName: \"kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8\") pod \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " Nov 01 15:41:28.584147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.583935 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home\") pod \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " Nov 01 15:41:28.584147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.583977 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir\") pod \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " Nov 01 15:41:28.584348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.584155 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache" pod "4e52a4d1-8264-4b3f-980e-971cf6ca16cd" (UID: "4e52a4d1-8264-4b3f-980e-971cf6ca16cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:28.584348 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.584202 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home" pod "4e52a4d1-8264-4b3f-980e-971cf6ca16cd" (UID: "4e52a4d1-8264-4b3f-980e-971cf6ca16cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:28.584444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.584352 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir" pod "4e52a4d1-8264-4b3f-980e-971cf6ca16cd" (UID: "4e52a4d1-8264-4b3f-980e-971cf6ca16cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:28.584530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.584507 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm\") pod \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\" (UID: \"4e52a4d1-8264-4b3f-980e-971cf6ca16cd\") " Nov 01 15:41:28.584989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.584967 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.585075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.584996 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.585075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.585013 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.586285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.586259 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs" pod "4e52a4d1-8264-4b3f-980e-971cf6ca16cd" (UID: "4e52a4d1-8264-4b3f-980e-971cf6ca16cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:41:28.586390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.586281 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8" pod "4e52a4d1-8264-4b3f-980e-971cf6ca16cd" (UID: "4e52a4d1-8264-4b3f-980e-971cf6ca16cd"). InnerVolumeSpecName "kube-api-access-w2jg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:28.586440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.586408 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm" pod "4e52a4d1-8264-4b3f-980e-971cf6ca16cd" (UID: "4e52a4d1-8264-4b3f-980e-971cf6ca16cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:28.686050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.686013 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.686050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.686044 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jg8\" (UniqueName: \"kubernetes.io/projected/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-kube-api-access-w2jg8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.686050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.686054 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e52a4d1-8264-4b3f-980e-971cf6ca16cd-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:28.760903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.760855 5473 generic.go:356] "Generic (PLEG): container finished" podID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerID="ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f" exitCode=0 Nov 01 15:41:28.761097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.760963 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" Nov 01 15:41:28.761097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.760971 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" event={"ID":"abe101fc-9a38-4bcf-bf63-002bbf93709d","Type":"ContainerDied","Data":"ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f"} Nov 01 15:41:28.761097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.761042 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg" event={"ID":"abe101fc-9a38-4bcf-bf63-002bbf93709d","Type":"ContainerDied","Data":"a62fc2024ec0b708a20c081b0ddd3420e29d412aa81c48a3148e6e6ce481a1ba"} Nov 01 15:41:28.761097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.761075 5473 scope.go:122] "RemoveContainer" containerID="ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f" Nov 01 15:41:28.762606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.762576 5473 generic.go:356] "Generic (PLEG): container finished" podID="94500465-cb49-41e5-9378-71b86396fda9" containerID="de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93" exitCode=0 Nov 01 15:41:28.762717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.762614 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" event={"ID":"94500465-cb49-41e5-9378-71b86396fda9","Type":"ContainerDied","Data":"de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93"} Nov 01 15:41:28.762717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.762638 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" Nov 01 15:41:28.762717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.762664 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq" event={"ID":"94500465-cb49-41e5-9378-71b86396fda9","Type":"ContainerDied","Data":"2b9164b48f6126086d204251c672895ee3ceec9f0850a47677cd790cee2065ad"} Nov 01 15:41:28.764414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.764393 5473 generic.go:356] "Generic (PLEG): container finished" podID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerID="a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a" exitCode=0 Nov 01 15:41:28.764536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.764451 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" Nov 01 15:41:28.764536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.764491 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" event={"ID":"4e52a4d1-8264-4b3f-980e-971cf6ca16cd","Type":"ContainerDied","Data":"a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a"} Nov 01 15:41:28.764680 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.764541 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm" event={"ID":"4e52a4d1-8264-4b3f-980e-971cf6ca16cd","Type":"ContainerDied","Data":"6fc052490a8f83378a366a5d5a3994652642705852c1c6b7d171964f24117138"} Nov 01 15:41:28.766239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.766205 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" Nov 01 15:41:28.766438 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.766206 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88" event={"ID":"0f633d5b-4ebc-4718-b1ee-2e0f33baba07","Type":"ContainerDied","Data":"12a69b6da22dc842fc54b104c5f3b93deafde07d403bf65879d588f6d5e8ab6e"} Nov 01 15:41:28.766530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.766455 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a69b6da22dc842fc54b104c5f3b93deafde07d403bf65879d588f6d5e8ab6e" Nov 01 15:41:28.778096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.778070 5473 scope.go:122] "RemoveContainer" containerID="ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f" Nov 01 15:41:28.780836 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:28.780805 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f\": container with ID starting with ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f not found: ID does not exist" containerID="ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f" Nov 01 15:41:28.780947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.780842 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f"} err="failed to get container status \"ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f\": rpc error: code = NotFound desc = could not find container \"ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f\": container with ID starting with ae35425bc33e5c309b1484a3705217efa1c5111a8610c89faa77dca0f4ab078f not found: ID does not exist" Nov 01 15:41:28.780947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.780863 5473 scope.go:122] "RemoveContainer" containerID="de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93" Nov 01 15:41:28.792025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.791992 5473 scope.go:122] "RemoveContainer" containerID="de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93" Nov 01 15:41:28.792380 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:28.792357 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93\": container with ID starting with de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93 not found: ID does not exist" containerID="de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93" Nov 01 15:41:28.792460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.792390 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93"} err="failed to get container status \"de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93\": rpc error: code = NotFound desc = could not find container \"de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93\": container with ID starting with de86888749f1e454bf96ea0e0aade5368150b7063c5087a374f46c6d83667a93 not found: ID does not exist" Nov 01 15:41:28.792460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.792407 5473 scope.go:122] "RemoveContainer" containerID="a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a" Nov 01 15:41:28.803129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.803105 5473 scope.go:122] "RemoveContainer" containerID="a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a" Nov 01 15:41:28.803479 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:28.803454 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a\": container with ID starting with a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a not found: ID does not exist" containerID="a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a" Nov 01 15:41:28.803595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.803485 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a"} err="failed to get container status \"a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a\": rpc error: code = NotFound desc = could not find container \"a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a\": container with ID starting with a6fbf59b2a9d2b160deb620b03e086ae08fd407ca8f572a7716f0b8170b2e63a not found: ID does not exist" Nov 01 15:41:28.929294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.929242 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq"] Nov 01 15:41:28.939208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:28.939172 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96165a-85dd4b67ff-kt7vq"] Nov 01 15:41:29.081830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.081796 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94500465-cb49-41e5-9378-71b86396fda9" path="/var/lib/kubelet/pods/94500465-cb49-41e5-9378-71b86396fda9/volumes" Nov 01 15:41:29.103327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.103283 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg"] Nov 01 15:41:29.106382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.106361 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96165a-7445bb8945-j7lrg"] Nov 01 15:41:29.215198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.215111 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm"] Nov 01 15:41:29.220495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.220467 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-007270/premium-w2-007270-kserve-666c6ff7d-rr6dm"] Nov 01 15:41:29.484433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.484353 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:29.786561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.786526 5473 generic.go:356] "Generic (PLEG): container finished" podID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerID="55591be9f016706e984e911b34cc6faa497a1c1ca775b220c9080d4acaeb205a" exitCode=0 Nov 01 15:41:29.786711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.786581 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" event={"ID":"88f9ff98-2bbc-4126-b236-9bcc598d02d6","Type":"ContainerDied","Data":"55591be9f016706e984e911b34cc6faa497a1c1ca775b220c9080d4acaeb205a"} Nov 01 15:41:29.786711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.786645 5473 scope.go:122] "RemoveContainer" containerID="2358c93315e86b82a3574abc11ed577a7be55c205df05bc2cb7e36b5580545a1" Nov 01 15:41:29.800542 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.800522 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:41:29.897330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897298 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897350 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897420 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897484 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897515 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897568 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897611 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897656 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897722 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy\") pod \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\" (UID: \"88f9ff98-2bbc-4126-b236-9bcc598d02d6\") " Nov 01 15:41:29.897787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897734 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:29.898070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897840 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:29.898070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897875 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:29.898070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.897978 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:29.898282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.898107 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:29.898349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.898308 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.898349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.898329 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.898349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.898341 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.898349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.898349 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.898573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.898358 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.900024 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.899987 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:29.900275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.900252 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:29.900372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.900342 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "kube-api-access-nfvgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:29.900508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.900409 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo" pod "88f9ff98-2bbc-4126-b236-9bcc598d02d6" (UID: "88f9ff98-2bbc-4126-b236-9bcc598d02d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:41:29.999192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.999147 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvgz\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-kube-api-access-nfvgz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.999192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.999187 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.999192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.999200 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:29.999192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:29.999208 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88f9ff98-2bbc-4126-b236-9bcc598d02d6-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:30.794602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:30.794569 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" Nov 01 15:41:30.795111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:30.794619 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm" event={"ID":"88f9ff98-2bbc-4126-b236-9bcc598d02d6","Type":"ContainerDied","Data":"f8233e04f508a25490c2e4f8c88938da3c80f514814c372d35b7dcee2d8f3c5a"} Nov 01 15:41:30.795111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:30.794680 5473 scope.go:122] "RemoveContainer" containerID="55591be9f016706e984e911b34cc6faa497a1c1ca775b220c9080d4acaeb205a" Nov 01 15:41:31.026667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.026629 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm"] Nov 01 15:41:31.030406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.030374 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96165a-openshift-default-545b8bd88c-z9bcm"] Nov 01 15:41:31.086035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.085946 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" path="/var/lib/kubelet/pods/4e52a4d1-8264-4b3f-980e-971cf6ca16cd/volumes" Nov 01 15:41:31.086534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.086512 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" path="/var/lib/kubelet/pods/88f9ff98-2bbc-4126-b236-9bcc598d02d6/volumes" Nov 01 15:41:31.086928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.086885 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" path="/var/lib/kubelet/pods/abe101fc-9a38-4bcf-bf63-002bbf93709d/volumes" Nov 01 15:41:31.115524 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:31.115480 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:41:31.143471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.143441 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:31.304297 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.304267 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:31.439985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.439901 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerName="payload-processing" containerID="cri-o://9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253" gracePeriod=25 Nov 01 15:41:31.508746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.508699 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerName="payload-pre-processing" containerID="cri-o://31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5" gracePeriod=25 Nov 01 15:41:31.741493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.741399 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:31.890481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:31.890433 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:32.406352 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:32.406322 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef29d51_4057_4f6c_8533_c03c355a53db.slice/crio-conmon-dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 15:41:32.613950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.613882 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:41:32.730308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730276 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730331 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730371 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730403 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730478 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730696 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730512 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730696 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730595 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730698 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730721 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:32.730819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730732 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9\") pod \"6ef29d51-4057-4f6c-8533-c03c355a53db\" (UID: \"6ef29d51-4057-4f6c-8533-c03c355a53db\") " Nov 01 15:41:32.730819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730787 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:32.731039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730888 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:32.731039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.730907 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:32.731151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.731091 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:32.731319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.731287 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.731319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.731311 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.731319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.731322 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef29d51-4057-4f6c-8533-c03c355a53db-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.731583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.731331 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.731583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.731341 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.732940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.732915 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:41:32.733032 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.732924 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:32.733193 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.733173 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "kube-api-access-jhlh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:32.733273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.733179 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy" pod "6ef29d51-4057-4f6c-8533-c03c355a53db" (UID: "6ef29d51-4057-4f6c-8533-c03c355a53db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:41:32.821812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.821762 5473 generic.go:356] "Generic (PLEG): container finished" podID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerID="dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a" exitCode=0 Nov 01 15:41:32.822000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.821829 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" event={"ID":"6ef29d51-4057-4f6c-8533-c03c355a53db","Type":"ContainerDied","Data":"dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a"} Nov 01 15:41:32.822000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.821926 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" event={"ID":"6ef29d51-4057-4f6c-8533-c03c355a53db","Type":"ContainerDied","Data":"a23b0ddfd34da4053411c42fe087ae4f98ff041dbe4b2a1870663730b44bfeba"} Nov 01 15:41:32.822000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.821965 5473 scope.go:122] "RemoveContainer" containerID="dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a" Nov 01 15:41:32.822206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.822186 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8" Nov 01 15:41:32.832294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.832269 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.832410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.832298 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhlh9\" (UniqueName: \"kubernetes.io/projected/6ef29d51-4057-4f6c-8533-c03c355a53db-kube-api-access-jhlh9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.832410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.832314 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.832410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.832327 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef29d51-4057-4f6c-8533-c03c355a53db-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:32.834807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.834785 5473 scope.go:122] "RemoveContainer" containerID="3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006" Nov 01 15:41:32.844900 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.844877 5473 scope.go:122] "RemoveContainer" containerID="dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a" Nov 01 15:41:32.845184 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:32.845164 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a\": container with ID starting with dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a not found: ID does not exist" containerID="dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a" Nov 01 15:41:32.845270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.845191 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a"} err="failed to get container status \"dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a\": rpc error: code = NotFound desc = could not find container \"dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a\": container with ID starting with dff191e19ff143bb6cbbf1003935c2b8bbf640be44d70d64091dfbe4fef8774a not found: ID does not exist" Nov 01 15:41:32.845270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.845210 5473 scope.go:122] "RemoveContainer" containerID="3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006" Nov 01 15:41:32.845481 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:32.845464 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006\": container with ID starting with 3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006 not found: ID does not exist" containerID="3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006" Nov 01 15:41:32.845532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.845486 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006"} err="failed to get container status \"3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006\": rpc error: code = NotFound desc = could not find container \"3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006\": container with ID starting with 3b7ae3a8e22a111cc4568340222fd318e868f736672a7a38fc1b3bbda0b72006 not found: ID does not exist" Nov 01 15:41:32.987817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.987718 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8"] Nov 01 15:41:32.991118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:32.991090 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-55943a-openshift-default-dccf6bdf-j6ws8"] Nov 01 15:41:33.081199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:33.081168 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" path="/var/lib/kubelet/pods/6ef29d51-4057-4f6c-8533-c03c355a53db/volumes" Nov 01 15:41:33.905279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:33.905186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:34.979019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:34.978984 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:35.175977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:35.175940 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk"] Nov 01 15:41:35.281603 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:35.281565 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7"] Nov 01 15:41:35.540233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:35.540123 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:35.693834 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:35.693790 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:35.986798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:35.986713 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:36.625550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:36.625516 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:37.048928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:37.048894 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:37.723563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:37.723534 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:38.542356 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:38.542315 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:38.714430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:38.714391 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:39.591703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:39.591656 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:40.176773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:40.176729 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" podUID="164930f3-c671-4958-9498-58f04f270361" containerName="payload-processing" containerID="cri-o://a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4" gracePeriod=25 Nov 01 15:41:40.283241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:40.283166 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerName="payload-pre-processing" containerID="cri-o://a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967" gracePeriod=25 Nov 01 15:41:41.185495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:41.185453 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:42.824059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:42.824020 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:42.824481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:42.824022 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" podUID="164930f3-c671-4958-9498-58f04f270361" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:43.675729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:43.675691 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:43.922431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:43.922378 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:44.847483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:44.847450 5473 scope.go:122] "RemoveContainer" containerID="89953ae20769c9b9b3f8acea9cde5703730aca434acada17c6b9d395fa2e8e5a" Nov 01 15:41:44.978019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:44.977980 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:45.588816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:45.588783 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:45.986547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:45.986460 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:48.592345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:48.592308 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:48.765586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:48.765543 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:49.779148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:49.779114 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:49.943602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:49.943567 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:50.118205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:50.118124 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:50.712884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:50.712850 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:51.751928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.751899 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 15:41:51.788189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.788167 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 15:41:51.832864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.832834 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume\") pod \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " Nov 01 15:41:51.833039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.832905 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4\" (UniqueName: \"kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4\") pod \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\" (UID: \"1cc8104c-bf33-4a46-9d04-0184d59e07c1\") " Nov 01 15:41:51.833039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.832926 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt\" (UniqueName: \"kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt\") pod \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " Nov 01 15:41:51.833039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.833011 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume\") pod \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\" (UID: \"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23\") " Nov 01 15:41:51.833363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.833336 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume" pod "1cc8104c-bf33-4a46-9d04-0184d59e07c1" (UID: "1cc8104c-bf33-4a46-9d04-0184d59e07c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:51.833477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.833440 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume" pod "9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" (UID: "9f7dd0e6-ac53-4ebb-8b61-60dad4575c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:41:51.835034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.835009 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt" pod "9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" (UID: "9f7dd0e6-ac53-4ebb-8b61-60dad4575c23"). InnerVolumeSpecName "kube-api-access-v6kkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:51.835121 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.835038 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4" pod "1cc8104c-bf33-4a46-9d04-0184d59e07c1" (UID: "1cc8104c-bf33-4a46-9d04-0184d59e07c1"). InnerVolumeSpecName "kube-api-access-4ggt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:41:51.934074 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.934039 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ggt4\" (UniqueName: \"kubernetes.io/projected/1cc8104c-bf33-4a46-9d04-0184d59e07c1-kube-api-access-4ggt4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:51.934074 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.934068 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kkt\" (UniqueName: \"kubernetes.io/projected/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-kube-api-access-v6kkt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:51.934074 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.934080 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:51.934375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.934089 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cc8104c-bf33-4a46-9d04-0184d59e07c1-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:41:51.948889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.948856 5473 generic.go:356] "Generic (PLEG): container finished" podID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerID="31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5" exitCode=0 Nov 01 15:41:51.949028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.948906 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" Nov 01 15:41:51.949028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.948940 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" event={"ID":"1cc8104c-bf33-4a46-9d04-0184d59e07c1","Type":"ContainerDied","Data":"31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5"} Nov 01 15:41:51.949028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.948994 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7" event={"ID":"1cc8104c-bf33-4a46-9d04-0184d59e07c1","Type":"ContainerDied","Data":"aa4f9a4f6d2ec13392c660f9d0fe480a44e2156f9ccbe991d7f37d162058847b"} Nov 01 15:41:51.949028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.949023 5473 scope.go:122] "RemoveContainer" containerID="31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5" Nov 01 15:41:51.950533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.950510 5473 generic.go:356] "Generic (PLEG): container finished" podID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerID="9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253" exitCode=0 Nov 01 15:41:51.950645 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.950567 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" Nov 01 15:41:51.950645 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.950609 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" event={"ID":"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23","Type":"ContainerDied","Data":"9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253"} Nov 01 15:41:51.950731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.950659 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv" event={"ID":"9f7dd0e6-ac53-4ebb-8b61-60dad4575c23","Type":"ContainerDied","Data":"7aaeb5433a2a0c439172a1b0adcb3cc9b90fca454bca76471cdabfe7dd2c0097"} Nov 01 15:41:51.959354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.959288 5473 scope.go:122] "RemoveContainer" containerID="31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5" Nov 01 15:41:51.959633 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:51.959602 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5\": container with ID starting with 31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5 not found: ID does not exist" containerID="31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5" Nov 01 15:41:51.959723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.959631 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5"} err="failed to get container status \"31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5\": rpc error: code = NotFound desc = could not find container \"31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5\": container with ID starting with 31b95337f04921bf9d64692fff46aa5c5d4bbf4a0681a538917fc6245e3315a5 not found: ID does not exist" Nov 01 15:41:51.959723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.959648 5473 scope.go:122] "RemoveContainer" containerID="9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253" Nov 01 15:41:51.972136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.972019 5473 scope.go:122] "RemoveContainer" containerID="9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253" Nov 01 15:41:51.972364 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:41:51.972340 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253\": container with ID starting with 9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253 not found: ID does not exist" containerID="9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253" Nov 01 15:41:51.972448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:51.972369 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253"} err="failed to get container status \"9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253\": rpc error: code = NotFound desc = could not find container \"9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253\": container with ID starting with 9a910870def74454d60fdc2899be396c6445406db5497ecf1fbeccb75d353253 not found: ID does not exist" Nov 01 15:41:52.092059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:52.092028 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7"] Nov 01 15:41:52.094885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:52.094859 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-007270-7d486cd587-ngnv7"] Nov 01 15:41:52.205175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:52.205101 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv"] Nov 01 15:41:52.209373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:52.209349 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-007270-5dcbdfc9c5-4cmdv"] Nov 01 15:41:52.824619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:52.824584 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:52.825016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:52.824632 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" podUID="164930f3-c671-4958-9498-58f04f270361" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:41:53.080815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:53.080730 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" path="/var/lib/kubelet/pods/1cc8104c-bf33-4a46-9d04-0184d59e07c1/volumes" Nov 01 15:41:53.081257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:53.081234 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" path="/var/lib/kubelet/pods/9f7dd0e6-ac53-4ebb-8b61-60dad4575c23/volumes" Nov 01 15:41:54.386940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:54.386906 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:56.149854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:56.149819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:41:58.793454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:41:58.793418 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:00.442716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.442640 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:42:00.521475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.521454 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn\" (UniqueName: \"kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn\") pod \"164930f3-c671-4958-9498-58f04f270361\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " Nov 01 15:42:00.521594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.521549 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume\") pod \"164930f3-c671-4958-9498-58f04f270361\" (UID: \"164930f3-c671-4958-9498-58f04f270361\") " Nov 01 15:42:00.521975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.521948 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume" pod "164930f3-c671-4958-9498-58f04f270361" (UID: "164930f3-c671-4958-9498-58f04f270361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:42:00.523532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.523512 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn" pod "164930f3-c671-4958-9498-58f04f270361" (UID: "164930f3-c671-4958-9498-58f04f270361"). InnerVolumeSpecName "kube-api-access-j8ldn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:00.538434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.538412 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:42:00.623010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.622957 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh\" (UniqueName: \"kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh\") pod \"8285dada-373e-45fc-8a1b-1a9d1530a98e\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " Nov 01 15:42:00.623247 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.623139 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume\") pod \"8285dada-373e-45fc-8a1b-1a9d1530a98e\" (UID: \"8285dada-373e-45fc-8a1b-1a9d1530a98e\") " Nov 01 15:42:00.623481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.623461 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ldn\" (UniqueName: \"kubernetes.io/projected/164930f3-c671-4958-9498-58f04f270361-kube-api-access-j8ldn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:00.623545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.623485 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164930f3-c671-4958-9498-58f04f270361-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:00.623545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.623506 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume" pod "8285dada-373e-45fc-8a1b-1a9d1530a98e" (UID: "8285dada-373e-45fc-8a1b-1a9d1530a98e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:42:00.625087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.625065 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh" pod "8285dada-373e-45fc-8a1b-1a9d1530a98e" (UID: "8285dada-373e-45fc-8a1b-1a9d1530a98e"). InnerVolumeSpecName "kube-api-access-q2knh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:00.724957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.724871 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8285dada-373e-45fc-8a1b-1a9d1530a98e-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:00.724957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:00.724903 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2knh\" (UniqueName: \"kubernetes.io/projected/8285dada-373e-45fc-8a1b-1a9d1530a98e-kube-api-access-q2knh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:01.009531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.009436 5473 generic.go:356] "Generic (PLEG): container finished" podID="164930f3-c671-4958-9498-58f04f270361" containerID="a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4" exitCode=0 Nov 01 15:42:01.009531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.009486 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" Nov 01 15:42:01.009788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.009522 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" event={"ID":"164930f3-c671-4958-9498-58f04f270361","Type":"ContainerDied","Data":"a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4"} Nov 01 15:42:01.009788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.009588 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk" event={"ID":"164930f3-c671-4958-9498-58f04f270361","Type":"ContainerDied","Data":"7de82e051e6f3c03180ba46660b2830c673e78f11ae21820fdddf2556b8071c1"} Nov 01 15:42:01.009788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.009617 5473 scope.go:122] "RemoveContainer" containerID="a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4" Nov 01 15:42:01.011006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.010982 5473 generic.go:356] "Generic (PLEG): container finished" podID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerID="a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967" exitCode=0 Nov 01 15:42:01.011126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.011031 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" Nov 01 15:42:01.011126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.011052 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" event={"ID":"8285dada-373e-45fc-8a1b-1a9d1530a98e","Type":"ContainerDied","Data":"a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967"} Nov 01 15:42:01.011126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.011083 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7" event={"ID":"8285dada-373e-45fc-8a1b-1a9d1530a98e","Type":"ContainerDied","Data":"acd5c76ce0d27f7bcd13fbb8ed5d81e60b86bc24771d1bb31ad7832d1970b32a"} Nov 01 15:42:01.020988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.020924 5473 scope.go:122] "RemoveContainer" containerID="a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4" Nov 01 15:42:01.021339 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:01.021315 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4\": container with ID starting with a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4 not found: ID does not exist" containerID="a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4" Nov 01 15:42:01.021413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.021349 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4"} err="failed to get container status \"a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4\": rpc error: code = NotFound desc = could not find container \"a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4\": container with ID starting with a40e6466ac675583e5bfacdc43c1b31950d6a2b9c65b5fc7bf03ff0302f02bc4 not found: ID does not exist" Nov 01 15:42:01.021413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.021371 5473 scope.go:122] "RemoveContainer" containerID="a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967" Nov 01 15:42:01.030133 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.030114 5473 scope.go:122] "RemoveContainer" containerID="a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967" Nov 01 15:42:01.030430 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:01.030408 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967\": container with ID starting with a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967 not found: ID does not exist" containerID="a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967" Nov 01 15:42:01.030505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.030436 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967"} err="failed to get container status \"a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967\": rpc error: code = NotFound desc = could not find container \"a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967\": container with ID starting with a73d80c544c664cea8588bd70c7dcad41f0ac084e88a6f9bdd749ca26bce2967 not found: ID does not exist" Nov 01 15:42:01.109341 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:01.109307 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:42:01.173959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.173929 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk"] Nov 01 15:42:01.223845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.223813 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-af2dd2-7b8889987-dlntk"] Nov 01 15:42:01.285196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.285166 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7"] Nov 01 15:42:01.289359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.289333 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-af2dd2-746f49597f-fnrf7"] Nov 01 15:42:01.727766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:01.727684 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:02.679839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:02.679803 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:03.081137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:03.081102 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164930f3-c671-4958-9498-58f04f270361" path="/var/lib/kubelet/pods/164930f3-c671-4958-9498-58f04f270361/volumes" Nov 01 15:42:03.081672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:03.081656 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" path="/var/lib/kubelet/pods/8285dada-373e-45fc-8a1b-1a9d1530a98e/volumes" Nov 01 15:42:03.727072 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:03.727032 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:05.366244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:05.366186 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:09.111271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:09.111210 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:09.977452 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:09.977414 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:10.865535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:10.865500 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:11.238873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:11.238794 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:11.399088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:11.399052 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:12.408015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:12.407978 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:12.607245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:12.607192 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:12.779574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:12.779541 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:13.186631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:13.186508 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:13.904118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:13.904085 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:14.084814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:14.084779 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:14.437090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:14.437056 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:15.165721 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:15.165682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:15.524583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:15.524555 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:15.865027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:15.864947 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:17.729156 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:17.729123 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:19.076740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:19.076704 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:19.247918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:19.247889 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:19.852010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:19.851968 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:20.154285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:20.154187 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:20.892716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:20.892681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:21.052593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:21.052564 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:21.336985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:21.336952 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:21.759886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:21.759797 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l"] Nov 01 15:42:22.017893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:22.017861 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:22.286904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:22.286820 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:22.523080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:22.523049 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:22.707129 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:22.707053 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:24.255969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:24.255933 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:24.414544 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:24.414510 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:24.942730 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:24.942692 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:25.306389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:25.306356 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:26.760821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:26.760770 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerName="payload-pre-processing" containerID="cri-o://ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c" gracePeriod=25 Nov 01 15:42:26.770298 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:26.770271 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:29.290594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:29.290564 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:31.107053 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:31.107017 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:42:31.531593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:31.531558 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:42:33.697775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:33.697738 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:37.550765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:37.550731 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:38.660154 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:38.660121 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:40.299903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:40.299867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:40.588495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:40.588397 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:40.752050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:40.752016 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:41.531478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:41.531441 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:42:42.053139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:42.053101 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:42.477879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:42.477784 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs"] Nov 01 15:42:42.478070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:42.478042 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" containerID="cri-o://7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3" gracePeriod=30 Nov 01 15:42:42.505796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:42.505768 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:43.239310 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:43.239277 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:44.718101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:44.718069 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:42:44.743510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:44.743484 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:42:44.992839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:44.992775 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:47.017570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.017494 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 15:42:47.098350 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.098317 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume\") pod \"000947b6-3732-41f9-ba1b-86f344ac0d2a\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " Nov 01 15:42:47.098549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.098376 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5\" (UniqueName: \"kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5\") pod \"000947b6-3732-41f9-ba1b-86f344ac0d2a\" (UID: \"000947b6-3732-41f9-ba1b-86f344ac0d2a\") " Nov 01 15:42:47.098733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.098683 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume" pod "000947b6-3732-41f9-ba1b-86f344ac0d2a" (UID: "000947b6-3732-41f9-ba1b-86f344ac0d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:42:47.099376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.099351 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/000947b6-3732-41f9-ba1b-86f344ac0d2a-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.100651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.100625 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5" pod "000947b6-3732-41f9-ba1b-86f344ac0d2a" (UID: "000947b6-3732-41f9-ba1b-86f344ac0d2a"). InnerVolumeSpecName "kube-api-access-p74p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:47.200504 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.200459 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p74p5\" (UniqueName: \"kubernetes.io/projected/000947b6-3732-41f9-ba1b-86f344ac0d2a-kube-api-access-p74p5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.316629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.316530 5473 generic.go:356] "Generic (PLEG): container finished" podID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerID="ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c" exitCode=0 Nov 01 15:42:47.316629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.316577 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" Nov 01 15:42:47.316858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.316625 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" event={"ID":"000947b6-3732-41f9-ba1b-86f344ac0d2a","Type":"ContainerDied","Data":"ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c"} Nov 01 15:42:47.316858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.316676 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l" event={"ID":"000947b6-3732-41f9-ba1b-86f344ac0d2a","Type":"ContainerDied","Data":"0ce8f1ffd60f5ac20868291c868477cd9a8304cb8e7f6df16dc3088dacf920ea"} Nov 01 15:42:47.316858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.316703 5473 scope.go:122] "RemoveContainer" containerID="ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c" Nov 01 15:42:47.330000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.329977 5473 scope.go:122] "RemoveContainer" containerID="ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c" Nov 01 15:42:47.330312 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:47.330293 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c\": container with ID starting with ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c not found: ID does not exist" containerID="ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c" Nov 01 15:42:47.330376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.330322 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c"} err="failed to get container status \"ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c\": rpc error: code = NotFound desc = could not find container \"ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c\": container with ID starting with ea2c2525c1a65ed789e4a67ea03286202c79a1107a1fa6e64870d87e4ec73e2c not found: ID does not exist" Nov 01 15:42:47.483078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.483040 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l"] Nov 01 15:42:47.486073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.486045 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97c87c-ddb86bd55-n855l"] Nov 01 15:42:47.716940 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.716918 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:42:47.806111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806082 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806120 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806165 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806185 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806254 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806322 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806380 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806420 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806484 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs\") pod \"43b62e39-2b10-41d3-80e5-a449d34d0f23\" (UID: \"43b62e39-2b10-41d3-80e5-a449d34d0f23\") " Nov 01 15:42:47.806684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806575 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:47.806850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806814 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:47.806850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806741 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:42:47.806850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806830 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:47.807036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806898 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:47.807036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806971 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.807036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.806991 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.807036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.807006 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b62e39-2b10-41d3-80e5-a449d34d0f23-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.807036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.807019 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.808524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.808489 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:47.808638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.808541 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:42:47.808751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.808733 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:47.809008 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.808993 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7" pod "43b62e39-2b10-41d3-80e5-a449d34d0f23" (UID: "43b62e39-2b10-41d3-80e5-a449d34d0f23"). InnerVolumeSpecName "kube-api-access-vvgr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:47.908147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.908058 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgr7\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-kube-api-access-vvgr7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.908147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.908091 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.908147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.908100 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.908147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.908109 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b62e39-2b10-41d3-80e5-a449d34d0f23-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:47.908147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:47.908118 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b62e39-2b10-41d3-80e5-a449d34d0f23-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:48.285123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.285086 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:48.326159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.326127 5473 generic.go:356] "Generic (PLEG): container finished" podID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerID="7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3" exitCode=0 Nov 01 15:42:48.326363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.326182 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" Nov 01 15:42:48.326363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.326227 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" event={"ID":"43b62e39-2b10-41d3-80e5-a449d34d0f23","Type":"ContainerDied","Data":"7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3"} Nov 01 15:42:48.326363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.326284 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs" event={"ID":"43b62e39-2b10-41d3-80e5-a449d34d0f23","Type":"ContainerDied","Data":"d2fb5673d4ef4a90af68a54cc68cc046710b7977dcd4a5cb48d0c576520bd6f1"} Nov 01 15:42:48.326363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.326309 5473 scope.go:122] "RemoveContainer" containerID="7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3" Nov 01 15:42:48.336803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.336782 5473 scope.go:122] "RemoveContainer" containerID="699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df" Nov 01 15:42:48.345619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.345599 5473 scope.go:122] "RemoveContainer" containerID="7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3" Nov 01 15:42:48.345860 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:48.345844 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3\": container with ID starting with 7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3 not found: ID does not exist" containerID="7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3" Nov 01 15:42:48.345915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.345865 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3"} err="failed to get container status \"7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3\": rpc error: code = NotFound desc = could not find container \"7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3\": container with ID starting with 7de64a4fc0003f358951cdeb6f99823db8125c02f571d94e4e685dc0ebb240b3 not found: ID does not exist" Nov 01 15:42:48.345915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.345881 5473 scope.go:122] "RemoveContainer" containerID="699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df" Nov 01 15:42:48.346093 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:42:48.346075 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df\": container with ID starting with 699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df not found: ID does not exist" containerID="699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df" Nov 01 15:42:48.346136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.346100 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df"} err="failed to get container status \"699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df\": rpc error: code = NotFound desc = could not find container \"699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df\": container with ID starting with 699f7b5e69568738ee78e305171ea01cda4e90981efd8a7b2f213ca93f22f7df not found: ID does not exist" Nov 01 15:42:48.540707 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.540614 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs"] Nov 01 15:42:48.568536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.568504 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:48.591731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:48.591697 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ac11c-openshift-default-86d7b548f7-dpwqs"] Nov 01 15:42:49.080811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:49.080777 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" path="/var/lib/kubelet/pods/000947b6-3732-41f9-ba1b-86f344ac0d2a/volumes" Nov 01 15:42:49.081258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:49.081245 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" path="/var/lib/kubelet/pods/43b62e39-2b10-41d3-80e5-a449d34d0f23/volumes" Nov 01 15:42:49.368190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:49.368106 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:49.528415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:49.528374 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:49.797258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:49.797198 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:50.217911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.217831 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl"] Nov 01 15:42:50.218163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.218116 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" containerID="cri-o://257955cceadfe33656c8fa77004b16439e8e1bca16d8d9e80c8dc1138e7dabf6" gracePeriod=30 Nov 01 15:42:50.250771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.250735 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:50.447801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.447762 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t"] Nov 01 15:42:50.546294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.546258 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk"] Nov 01 15:42:50.815871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.815789 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:50.970352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:50.970322 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:53.997136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:53.997102 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:54.885810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:54.885781 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:55.373486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.373449 5473 generic.go:356] "Generic (PLEG): container finished" podID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerID="257955cceadfe33656c8fa77004b16439e8e1bca16d8d9e80c8dc1138e7dabf6" exitCode=0 Nov 01 15:42:55.373877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.373492 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" event={"ID":"093f8eb7-53cb-4245-9a53-888b2f8c8b4c","Type":"ContainerDied","Data":"257955cceadfe33656c8fa77004b16439e8e1bca16d8d9e80c8dc1138e7dabf6"} Nov 01 15:42:55.373877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.373555 5473 scope.go:122] "RemoveContainer" containerID="7199649c39a12ee9165c54518af6622d80b959797167204c761c903ec54df7a4" Nov 01 15:42:55.448392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.448347 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerName="payload-processing" containerID="cri-o://01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1" gracePeriod=25 Nov 01 15:42:55.477476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.477420 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 15:42:55.546961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.546920 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerName="payload-pre-processing" containerID="cri-o://e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda" gracePeriod=25 Nov 01 15:42:55.583272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583241 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583356 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583446 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583417 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583447 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583489 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583522 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583553 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583584 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583659 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert\") pod \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\" (UID: \"093f8eb7-53cb-4245-9a53-888b2f8c8b4c\") " Nov 01 15:42:55.583899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583785 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:55.584021 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.583977 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:55.584131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.584070 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.584131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.584090 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.584277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.584176 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:42:55.584277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.584183 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:55.584277 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.584197 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:55.585916 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.585894 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:42:55.586022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.585906 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:42:55.586087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.586035 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:55.586087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.586065 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d" pod "093f8eb7-53cb-4245-9a53-888b2f8c8b4c" (UID: "093f8eb7-53cb-4245-9a53-888b2f8c8b4c"). InnerVolumeSpecName "kube-api-access-bz98d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:42:55.684905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684869 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.684905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684900 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.684905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684909 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.685144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684919 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.685144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684928 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.685144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684937 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:55.685144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:55.684945 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz98d\" (UniqueName: \"kubernetes.io/projected/093f8eb7-53cb-4245-9a53-888b2f8c8b4c-kube-api-access-bz98d\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:42:56.382478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.382440 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" event={"ID":"093f8eb7-53cb-4245-9a53-888b2f8c8b4c","Type":"ContainerDied","Data":"95674e1f48cf3b448ac186af8ab1351972c930990fc6c5e028a442779803f58e"} Nov 01 15:42:56.382478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.382467 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl" Nov 01 15:42:56.383146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.382523 5473 scope.go:122] "RemoveContainer" containerID="257955cceadfe33656c8fa77004b16439e8e1bca16d8d9e80c8dc1138e7dabf6" Nov 01 15:42:56.444673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.444635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:56.599414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.599378 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl"] Nov 01 15:42:56.606263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.606235 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97c87c-openshift-default-75d48cbb75-vxgkl"] Nov 01 15:42:56.606800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:56.606782 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:57.046769 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:57.046733 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:42:57.081148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:57.081117 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" path="/var/lib/kubelet/pods/093f8eb7-53cb-4245-9a53-888b2f8c8b4c/volumes" Nov 01 15:42:57.139362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:57.139323 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:42:57.139362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:57.139344 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:42:58.522960 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:42:58.522928 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:01.107134 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:01.107101 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:43:02.558705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:02.558667 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:02.724757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:02.724725 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:04.084484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:04.084450 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:07.139027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:07.138988 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:43:07.139524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:07.139064 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:43:07.928096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:07.928063 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:09.158947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:09.158916 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:09.327252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:09.327202 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:09.490106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:09.490021 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:10.310831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:10.310792 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:10.637632 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:10.637556 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:11.073638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:11.073607 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5"] Nov 01 15:43:11.073881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:11.073828 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" containerID="cri-o://dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653" gracePeriod=30 Nov 01 15:43:11.095646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:11.095622 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:11.532124 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:11.532091 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:11.862597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:11.862512 5473 patch_prober.go:29] interesting pod/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:43:11.862597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:11.862572 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:43:12.085912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.085881 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk"] Nov 01 15:43:12.086312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086298 5473 cpu_manager.go:414] "containerMap: removing container" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086314 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086328 5473 cpu_manager.go:414] "containerMap: removing container" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086334 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086346 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086352 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086360 5473 cpu_manager.go:414] "containerMap: removing container" podUID="94500465-cb49-41e5-9378-71b86396fda9" containerName="payload-pre-processing" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086366 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94500465-cb49-41e5-9378-71b86396fda9" containerName="payload-pre-processing" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086376 5473 cpu_manager.go:414] "containerMap: removing container" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerName="payload-pre-processing" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086381 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerName="payload-pre-processing" Nov 01 15:43:12.086386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086389 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086394 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086402 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086406 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086413 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086418 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086428 5473 cpu_manager.go:414] "containerMap: removing container" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086433 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086440 5473 cpu_manager.go:414] "containerMap: removing container" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086444 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086455 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086460 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086467 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086472 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086480 5473 cpu_manager.go:414] "containerMap: removing container" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086484 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086496 5473 cpu_manager.go:414] "containerMap: removing container" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086500 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086508 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0f633d5b-4ebc-4718-b1ee-2e0f33baba07" containerName="revoke-keys" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086513 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f633d5b-4ebc-4718-b1ee-2e0f33baba07" containerName="revoke-keys" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086534 5473 cpu_manager.go:414] "containerMap: removing container" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086541 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086550 5473 cpu_manager.go:414] "containerMap: removing container" podUID="164930f3-c671-4958-9498-58f04f270361" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086554 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164930f3-c671-4958-9498-58f04f270361" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086608 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="164930f3-c671-4958-9498-58f04f270361" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086615 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086620 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b62e39-2b10-41d3-80e5-a449d34d0f23" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086628 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ceda98-4397-4c78-91ed-655e5fe7523e" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086636 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086642 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="94500465-cb49-41e5-9378-71b86396fda9" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086648 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="000947b6-3732-41f9-ba1b-86f344ac0d2a" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086653 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc8104c-bf33-4a46-9d04-0184d59e07c1" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086660 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e52a4d1-8264-4b3f-980e-971cf6ca16cd" containerName="main" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086665 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef29d51-4057-4f6c-8533-c03c355a53db" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086671 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe101fc-9a38-4bcf-bf63-002bbf93709d" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086677 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086682 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7dd0e6-ac53-4ebb-8b61-60dad4575c23" containerName="payload-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086687 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8285dada-373e-45fc-8a1b-1a9d1530a98e" containerName="payload-pre-processing" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086692 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086699 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f633d5b-4ebc-4718-b1ee-2e0f33baba07" containerName="revoke-keys" Nov 01 15:43:12.086705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.086704 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f9ff98-2bbc-4126-b236-9bcc598d02d6" containerName="istio-proxy" Nov 01 15:43:12.091606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.091590 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.145458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.145392 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk"] Nov 01 15:43:12.192184 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.192156 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:43:12.192184 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.192160 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:43:12.236235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.236199 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7chl\" (UniqueName: \"kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.236358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.236271 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.236358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.236295 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.337057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.337027 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7chl\" (UniqueName: \"kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.337057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.337075 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.337270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.337099 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.337792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.337770 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.339383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.339365 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.393986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.393960 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7chl\" (UniqueName: \"kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl\") pod \"maas-api-revoke-sub-2c79895f-jftpk\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.407305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.407249 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:12.774323 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:43:12.774277 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb520751d_61b0_4410_9d09_18c3148a422f.slice/crio-d8e0cbd15412d1e96d27e7de8beb0aa09cdea388e2c7417ce83081b6bb0de2c6 WatchSource:0}: Error finding container d8e0cbd15412d1e96d27e7de8beb0aa09cdea388e2c7417ce83081b6bb0de2c6: Status 404 returned error can't find the container with id d8e0cbd15412d1e96d27e7de8beb0aa09cdea388e2c7417ce83081b6bb0de2c6 Nov 01 15:43:12.777674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:12.777644 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk"] Nov 01 15:43:13.498152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:13.498066 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" event={"ID":"b520751d-61b0-4410-9d09-18c3148a422f","Type":"ContainerStarted","Data":"8060f27a668e709a3afd10b8dcd46f44b1da2567c1161110927dcfeccf69b947"} Nov 01 15:43:13.498345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:13.498175 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" event={"ID":"b520751d-61b0-4410-9d09-18c3148a422f","Type":"ContainerStarted","Data":"d8e0cbd15412d1e96d27e7de8beb0aa09cdea388e2c7417ce83081b6bb0de2c6"} Nov 01 15:43:13.611377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:13.611325 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" podStartSLOduration=1.611307748 podStartE2EDuration="1.611307748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:43:13.610293058 +0000 UTC m=+5132.668444911" watchObservedRunningTime="2026-11-01 15:43:13.611307748 +0000 UTC m=+5132.669459599" Nov 01 15:43:14.318137 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:14.318104 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:14.506526 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:14.506493 5473 generic.go:356] "Generic (PLEG): container finished" podID="b520751d-61b0-4410-9d09-18c3148a422f" containerID="8060f27a668e709a3afd10b8dcd46f44b1da2567c1161110927dcfeccf69b947" exitCode=0 Nov 01 15:43:14.506709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:14.506575 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" event={"ID":"b520751d-61b0-4410-9d09-18c3148a422f","Type":"ContainerDied","Data":"8060f27a668e709a3afd10b8dcd46f44b1da2567c1161110927dcfeccf69b947"} Nov 01 15:43:15.681778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.681750 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:15.693813 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.693236 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:43:15.772212 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772181 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca\") pod \"b520751d-61b0-4410-9d09-18c3148a422f\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " Nov 01 15:43:15.772212 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772239 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume\") pod \"95b41ae1-1984-4103-8f6d-9308335bbc95\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " Nov 01 15:43:15.772529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772321 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl\" (UniqueName: \"kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl\") pod \"b520751d-61b0-4410-9d09-18c3148a422f\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " Nov 01 15:43:15.772529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772400 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn\") pod \"95b41ae1-1984-4103-8f6d-9308335bbc95\" (UID: \"95b41ae1-1984-4103-8f6d-9308335bbc95\") " Nov 01 15:43:15.772529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772435 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca\") pod \"b520751d-61b0-4410-9d09-18c3148a422f\" (UID: \"b520751d-61b0-4410-9d09-18c3148a422f\") " Nov 01 15:43:15.772732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772609 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca" pod "b520751d-61b0-4410-9d09-18c3148a422f" (UID: "b520751d-61b0-4410-9d09-18c3148a422f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:15.772732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772688 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume" pod "95b41ae1-1984-4103-8f6d-9308335bbc95" (UID: "95b41ae1-1984-4103-8f6d-9308335bbc95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:15.772817 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.772769 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:15.774609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.774584 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca" pod "b520751d-61b0-4410-9d09-18c3148a422f" (UID: "b520751d-61b0-4410-9d09-18c3148a422f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:43:15.774716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.774627 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn" pod "95b41ae1-1984-4103-8f6d-9308335bbc95" (UID: "95b41ae1-1984-4103-8f6d-9308335bbc95"). InnerVolumeSpecName "kube-api-access-x4pmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:15.774716 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.774655 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl" pod "b520751d-61b0-4410-9d09-18c3148a422f" (UID: "b520751d-61b0-4410-9d09-18c3148a422f"). InnerVolumeSpecName "kube-api-access-t7chl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:15.785126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.785101 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:43:15.874117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874087 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t\" (UniqueName: \"kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t\") pod \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " Nov 01 15:43:15.874250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874167 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume\") pod \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\" (UID: \"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6\") " Nov 01 15:43:15.874541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874515 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume" pod "73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" (UID: "73aa7734-f8c0-41ac-bfa9-a58918e1c6e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:15.874541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874534 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pmn\" (UniqueName: \"kubernetes.io/projected/95b41ae1-1984-4103-8f6d-9308335bbc95-kube-api-access-x4pmn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:15.874714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874553 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b520751d-61b0-4410-9d09-18c3148a422f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:15.874714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874564 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95b41ae1-1984-4103-8f6d-9308335bbc95-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:15.874714 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.874573 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7chl\" (UniqueName: \"kubernetes.io/projected/b520751d-61b0-4410-9d09-18c3148a422f-kube-api-access-t7chl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:15.876199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.876178 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t" pod "73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" (UID: "73aa7734-f8c0-41ac-bfa9-a58918e1c6e6"). InnerVolumeSpecName "kube-api-access-fxw7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:15.975958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.975869 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxw7t\" (UniqueName: \"kubernetes.io/projected/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-kube-api-access-fxw7t\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:15.975958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:15.975907 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.312534 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.312510 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:43:16.480918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.480878 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.480970 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481015 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481078 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481151 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481181 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481252 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481194 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:16.481491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481349 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481412 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:16.481491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481450 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.481491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481484 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token\") pod \"a1c04e05-8282-4efc-bf59-f6055d051095\" (UID: \"a1c04e05-8282-4efc-bf59-f6055d051095\") " Nov 01 15:43:16.482453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481666 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:16.482453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481813 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:16.482453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481953 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.482453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481974 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.482453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.481988 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.482453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.482001 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.482828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.482450 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:16.484097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.484069 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:43:16.484248 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.484145 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:16.484424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.484402 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "kube-api-access-5rwvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:16.484731 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.484693 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy" pod "a1c04e05-8282-4efc-bf59-f6055d051095" (UID: "a1c04e05-8282-4efc-bf59-f6055d051095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:16.485086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.485066 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh"] Nov 01 15:43:16.523433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.523409 5473 generic.go:356] "Generic (PLEG): container finished" podID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerID="e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda" exitCode=0 Nov 01 15:43:16.523556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.523464 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" Nov 01 15:43:16.523556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.523497 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" event={"ID":"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6","Type":"ContainerDied","Data":"e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda"} Nov 01 15:43:16.523697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.523564 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk" event={"ID":"73aa7734-f8c0-41ac-bfa9-a58918e1c6e6","Type":"ContainerDied","Data":"597175bfb1e5de9a6f4a5c2b09fdc7c38f3df7eafeec3f532d60a4724f1bd8b3"} Nov 01 15:43:16.523697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.523599 5473 scope.go:122] "RemoveContainer" containerID="e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda" Nov 01 15:43:16.524976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.524896 5473 generic.go:356] "Generic (PLEG): container finished" podID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerID="01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1" exitCode=0 Nov 01 15:43:16.524976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.524946 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" Nov 01 15:43:16.525100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.524969 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" event={"ID":"95b41ae1-1984-4103-8f6d-9308335bbc95","Type":"ContainerDied","Data":"01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1"} Nov 01 15:43:16.525100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.525014 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t" event={"ID":"95b41ae1-1984-4103-8f6d-9308335bbc95","Type":"ContainerDied","Data":"eee1c60e886f03f51dd1daf48cc5759237e67a0dd24c5394359503a0d76b3142"} Nov 01 15:43:16.526928 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.526905 5473 generic.go:356] "Generic (PLEG): container finished" podID="a1c04e05-8282-4efc-bf59-f6055d051095" containerID="dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653" exitCode=0 Nov 01 15:43:16.527054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.527039 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" event={"ID":"a1c04e05-8282-4efc-bf59-f6055d051095","Type":"ContainerDied","Data":"dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653"} Nov 01 15:43:16.527094 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.527080 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" event={"ID":"a1c04e05-8282-4efc-bf59-f6055d051095","Type":"ContainerDied","Data":"5846ec2f16cc89cb90b3e3054192aa0e2f73b1d20bf41f730d9c7c7f0bf02bc2"} Nov 01 15:43:16.527160 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.527149 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5" Nov 01 15:43:16.531255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.530811 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" Nov 01 15:43:16.531255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.530826 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk" event={"ID":"b520751d-61b0-4410-9d09-18c3148a422f","Type":"ContainerDied","Data":"d8e0cbd15412d1e96d27e7de8beb0aa09cdea388e2c7417ce83081b6bb0de2c6"} Nov 01 15:43:16.531255 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.530881 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8e0cbd15412d1e96d27e7de8beb0aa09cdea388e2c7417ce83081b6bb0de2c6" Nov 01 15:43:16.534896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.534879 5473 scope.go:122] "RemoveContainer" containerID="e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda" Nov 01 15:43:16.535176 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:16.535156 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda\": container with ID starting with e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda not found: ID does not exist" containerID="e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda" Nov 01 15:43:16.535258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.535184 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda"} err="failed to get container status \"e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda\": rpc error: code = NotFound desc = could not find container \"e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda\": container with ID starting with e2dfbfcf5636d49cfa4a0bf76a67ae35e72c4ed6e3dd87bb64250c6231c0ddda not found: ID does not exist" Nov 01 15:43:16.535258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.535203 5473 scope.go:122] "RemoveContainer" containerID="01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1" Nov 01 15:43:16.545319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.545302 5473 scope.go:122] "RemoveContainer" containerID="01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1" Nov 01 15:43:16.545563 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:16.545545 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1\": container with ID starting with 01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1 not found: ID does not exist" containerID="01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1" Nov 01 15:43:16.545611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.545572 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1"} err="failed to get container status \"01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1\": rpc error: code = NotFound desc = could not find container \"01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1\": container with ID starting with 01dd1eb72a2c50f37d85f8eeeea901f7ed1390411ba8cd0eae0f47deb3de65c1 not found: ID does not exist" Nov 01 15:43:16.545611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.545588 5473 scope.go:122] "RemoveContainer" containerID="dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653" Nov 01 15:43:16.553934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.553918 5473 scope.go:122] "RemoveContainer" containerID="cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16" Nov 01 15:43:16.562191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.562174 5473 scope.go:122] "RemoveContainer" containerID="dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653" Nov 01 15:43:16.562446 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:16.562429 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653\": container with ID starting with dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653 not found: ID does not exist" containerID="dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653" Nov 01 15:43:16.562540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.562454 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653"} err="failed to get container status \"dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653\": rpc error: code = NotFound desc = could not find container \"dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653\": container with ID starting with dbc3d82417a4548ff61a2ec23b930bc98e1dc114e91b87b75100ddca66676653 not found: ID does not exist" Nov 01 15:43:16.562540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.562472 5473 scope.go:122] "RemoveContainer" containerID="cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16" Nov 01 15:43:16.562715 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:16.562698 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16\": container with ID starting with cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16 not found: ID does not exist" containerID="cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16" Nov 01 15:43:16.562767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.562722 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16"} err="failed to get container status \"cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16\": rpc error: code = NotFound desc = could not find container \"cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16\": container with ID starting with cf072908b6dab4b9d3ffa4332bc8df5db3e53c5d8dfbad2a63e99a0653896a16 not found: ID does not exist" Nov 01 15:43:16.582735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.582713 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rwvf\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-kube-api-access-5rwvf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.582831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.582738 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c04e05-8282-4efc-bf59-f6055d051095-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.582831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.582752 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c04e05-8282-4efc-bf59-f6055d051095-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.582831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.582765 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c04e05-8282-4efc-bf59-f6055d051095-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.582831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.582779 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c04e05-8282-4efc-bf59-f6055d051095-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:16.627176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.627153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:16.711177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.711152 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t"] Nov 01 15:43:16.713671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.713649 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8fd363-5c877fbf6f-jzg8t"] Nov 01 15:43:16.821115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.821055 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5"] Nov 01 15:43:16.825200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.825179 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-8fd363-openshift-default-66fd4486d7-nr4c5"] Nov 01 15:43:16.985571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.985541 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk"] Nov 01 15:43:16.991457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:16.991430 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-8fd363-77799bddb7-jbnjk"] Nov 01 15:43:17.081439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:17.081375 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" path="/var/lib/kubelet/pods/73aa7734-f8c0-41ac-bfa9-a58918e1c6e6/volumes" Nov 01 15:43:17.081753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:17.081740 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" path="/var/lib/kubelet/pods/95b41ae1-1984-4103-8f6d-9308335bbc95/volumes" Nov 01 15:43:17.082135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:17.082124 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" path="/var/lib/kubelet/pods/a1c04e05-8282-4efc-bf59-f6055d051095/volumes" Nov 01 15:43:17.119363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:17.119326 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:17.355157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:17.355072 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:18.586362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:18.586324 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:18.804645 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:18.804606 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:18.972046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:18.971954 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:19.133258 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:19.133229 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:19.488673 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:19.488641 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:20.036570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036536 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk"] Nov 01 15:43:20.036965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036911 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b520751d-61b0-4410-9d09-18c3148a422f" containerName="revoke-keys" Nov 01 15:43:20.036965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036923 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b520751d-61b0-4410-9d09-18c3148a422f" containerName="revoke-keys" Nov 01 15:43:20.036965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036937 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" Nov 01 15:43:20.036965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036942 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036971 5473 cpu_manager.go:414] "containerMap: removing container" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerName="payload-pre-processing" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036977 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerName="payload-pre-processing" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036986 5473 cpu_manager.go:414] "containerMap: removing container" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerName="payload-processing" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.036990 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerName="payload-processing" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.037043 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b520751d-61b0-4410-9d09-18c3148a422f" containerName="revoke-keys" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.037053 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.037061 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.037068 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="093f8eb7-53cb-4245-9a53-888b2f8c8b4c" containerName="istio-proxy" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.037075 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="73aa7734-f8c0-41ac-bfa9-a58918e1c6e6" containerName="payload-pre-processing" Nov 01 15:43:20.037113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.037080 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b41ae1-1984-4103-8f6d-9308335bbc95" containerName="payload-processing" Nov 01 15:43:20.041821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.041805 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.096662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.096638 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk"] Nov 01 15:43:20.143383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.143359 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:43:20.143530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.143440 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:43:20.215996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.215970 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.216135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.216018 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hgb\" (UniqueName: \"kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.216135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.216076 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.317448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.317353 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.317448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.317393 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hgb\" (UniqueName: \"kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.317448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.317417 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.318033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.318011 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.319745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.319724 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.373691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.373662 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hgb\" (UniqueName: \"kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb\") pod \"maas-api-revoke-sub-b9343056-5bwsk\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:20.654756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:20.654677 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:21.020578 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:43:21.020447 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6281a8be_7835_47d1_8254_e43fc2f9559f.slice/crio-2668efcbf3c97aa78196377bbfe97f9912aeade05a985b73dd8e8300e1c8273d WatchSource:0}: Error finding container 2668efcbf3c97aa78196377bbfe97f9912aeade05a985b73dd8e8300e1c8273d: Status 404 returned error can't find the container with id 2668efcbf3c97aa78196377bbfe97f9912aeade05a985b73dd8e8300e1c8273d Nov 01 15:43:21.023257 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:21.023236 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk"] Nov 01 15:43:21.564022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:21.563983 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" event={"ID":"6281a8be-7835-47d1-8254-e43fc2f9559f","Type":"ContainerStarted","Data":"b30e7b2ef6230b40fcaf84f6e656e4c91e2ef4bed3e4da6e9b570bfbb884f443"} Nov 01 15:43:21.564451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:21.564042 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" event={"ID":"6281a8be-7835-47d1-8254-e43fc2f9559f","Type":"ContainerStarted","Data":"2668efcbf3c97aa78196377bbfe97f9912aeade05a985b73dd8e8300e1c8273d"} Nov 01 15:43:21.677896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:21.677805 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" podStartSLOduration=2.677790264 podStartE2EDuration="2.677790264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:43:21.675734432 +0000 UTC m=+5140.733886282" watchObservedRunningTime="2026-11-01 15:43:21.677790264 +0000 UTC m=+5140.735942114" Nov 01 15:43:22.572865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:22.572835 5473 generic.go:356] "Generic (PLEG): container finished" podID="6281a8be-7835-47d1-8254-e43fc2f9559f" containerID="b30e7b2ef6230b40fcaf84f6e656e4c91e2ef4bed3e4da6e9b570bfbb884f443" exitCode=0 Nov 01 15:43:22.573304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:22.572915 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" event={"ID":"6281a8be-7835-47d1-8254-e43fc2f9559f","Type":"ContainerDied","Data":"b30e7b2ef6230b40fcaf84f6e656e4c91e2ef4bed3e4da6e9b570bfbb884f443"} Nov 01 15:43:23.509435 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.509398 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:23.724147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.724125 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:23.853152 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.853118 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb\" (UniqueName: \"kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb\") pod \"6281a8be-7835-47d1-8254-e43fc2f9559f\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " Nov 01 15:43:23.853354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.853186 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca\") pod \"6281a8be-7835-47d1-8254-e43fc2f9559f\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " Nov 01 15:43:23.853354 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.853306 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca\") pod \"6281a8be-7835-47d1-8254-e43fc2f9559f\" (UID: \"6281a8be-7835-47d1-8254-e43fc2f9559f\") " Nov 01 15:43:23.853575 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.853549 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca" pod "6281a8be-7835-47d1-8254-e43fc2f9559f" (UID: "6281a8be-7835-47d1-8254-e43fc2f9559f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:23.853687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.853670 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:23.855382 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.855353 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb" pod "6281a8be-7835-47d1-8254-e43fc2f9559f" (UID: "6281a8be-7835-47d1-8254-e43fc2f9559f"). InnerVolumeSpecName "kube-api-access-k4hgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:23.855503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.855455 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca" pod "6281a8be-7835-47d1-8254-e43fc2f9559f" (UID: "6281a8be-7835-47d1-8254-e43fc2f9559f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:43:23.954961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.954927 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6281a8be-7835-47d1-8254-e43fc2f9559f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:23.954961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:23.954953 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4hgb\" (UniqueName: \"kubernetes.io/projected/6281a8be-7835-47d1-8254-e43fc2f9559f-kube-api-access-k4hgb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:24.409134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.409097 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:24.431618 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.431590 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg"] Nov 01 15:43:24.468798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.468770 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh"] Nov 01 15:43:24.469251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.469231 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6281a8be-7835-47d1-8254-e43fc2f9559f" containerName="revoke-keys" Nov 01 15:43:24.469360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.469252 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6281a8be-7835-47d1-8254-e43fc2f9559f" containerName="revoke-keys" Nov 01 15:43:24.469360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.469304 5473 cpu_manager.go:414] "containerMap: removing container" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" Nov 01 15:43:24.469360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.469314 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c04e05-8282-4efc-bf59-f6055d051095" containerName="istio-proxy" Nov 01 15:43:24.469530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.469447 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6281a8be-7835-47d1-8254-e43fc2f9559f" containerName="revoke-keys" Nov 01 15:43:24.472581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.472563 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.575873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.575839 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh"] Nov 01 15:43:24.578333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.578306 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:24.588798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.588777 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" Nov 01 15:43:24.588931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.588809 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk" event={"ID":"6281a8be-7835-47d1-8254-e43fc2f9559f","Type":"ContainerDied","Data":"2668efcbf3c97aa78196377bbfe97f9912aeade05a985b73dd8e8300e1c8273d"} Nov 01 15:43:24.588931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.588864 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2668efcbf3c97aa78196377bbfe97f9912aeade05a985b73dd8e8300e1c8273d" Nov 01 15:43:24.660946 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.660885 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntzvs\" (UniqueName: \"kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.661060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.661003 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.661060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.661033 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.761556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.761524 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.762008 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.761560 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.762008 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.761636 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntzvs\" (UniqueName: \"kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.762192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.762170 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.763904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.763879 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:24.818328 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:24.818300 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntzvs\" (UniqueName: \"kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs\") pod \"maas-api-revoke-sub-6854c759-gkzhh\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:25.086327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:25.086294 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:25.272497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:25.272461 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442"] Nov 01 15:43:25.488588 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:43:25.488548 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45353db_a197_4986_875c_2f4e39564e28.slice/crio-e83a5d92affad687865ae8e0d81deefa08ce1f4c003be479f353a4a0dbcafb99 WatchSource:0}: Error finding container e83a5d92affad687865ae8e0d81deefa08ce1f4c003be479f353a4a0dbcafb99: Status 404 returned error can't find the container with id e83a5d92affad687865ae8e0d81deefa08ce1f4c003be479f353a4a0dbcafb99 Nov 01 15:43:25.491570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:25.491543 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh"] Nov 01 15:43:25.559370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:25.559337 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:25.597761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:25.597723 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" event={"ID":"f45353db-a197-4986-875c-2f4e39564e28","Type":"ContainerStarted","Data":"e83a5d92affad687865ae8e0d81deefa08ce1f4c003be479f353a4a0dbcafb99"} Nov 01 15:43:25.711592 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:25.711562 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:26.432411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:26.432383 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:26.604750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:26.604679 5473 generic.go:356] "Generic (PLEG): container finished" podID="f45353db-a197-4986-875c-2f4e39564e28" containerID="d49f3d1f56197c8fd578437d3326089bb064fb75fe029aa7d76dbb1823a6dc6b" exitCode=0 Nov 01 15:43:26.604889 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:26.604752 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" event={"ID":"f45353db-a197-4986-875c-2f4e39564e28","Type":"ContainerDied","Data":"d49f3d1f56197c8fd578437d3326089bb064fb75fe029aa7d76dbb1823a6dc6b"} Nov 01 15:43:27.747038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.747013 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:27.786811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.786785 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca\") pod \"f45353db-a197-4986-875c-2f4e39564e28\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " Nov 01 15:43:27.786984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.786860 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca\") pod \"f45353db-a197-4986-875c-2f4e39564e28\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " Nov 01 15:43:27.786984 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.786902 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs\" (UniqueName: \"kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs\") pod \"f45353db-a197-4986-875c-2f4e39564e28\" (UID: \"f45353db-a197-4986-875c-2f4e39564e28\") " Nov 01 15:43:27.787155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.787127 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca" pod "f45353db-a197-4986-875c-2f4e39564e28" (UID: "f45353db-a197-4986-875c-2f4e39564e28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:27.787309 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.787289 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45353db-a197-4986-875c-2f4e39564e28-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:27.788733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.788708 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca" pod "f45353db-a197-4986-875c-2f4e39564e28" (UID: "f45353db-a197-4986-875c-2f4e39564e28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:43:27.788876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.788850 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs" pod "f45353db-a197-4986-875c-2f4e39564e28" (UID: "f45353db-a197-4986-875c-2f4e39564e28"). InnerVolumeSpecName "kube-api-access-ntzvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:27.888802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.888724 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45353db-a197-4986-875c-2f4e39564e28-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:27.888802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:27.888765 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntzvs\" (UniqueName: \"kubernetes.io/projected/f45353db-a197-4986-875c-2f4e39564e28-kube-api-access-ntzvs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:28.262006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:28.261978 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:28.617136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:28.617056 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" Nov 01 15:43:28.617136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:28.617077 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh" event={"ID":"f45353db-a197-4986-875c-2f4e39564e28","Type":"ContainerDied","Data":"e83a5d92affad687865ae8e0d81deefa08ce1f4c003be479f353a4a0dbcafb99"} Nov 01 15:43:28.617136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:28.617120 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e83a5d92affad687865ae8e0d81deefa08ce1f4c003be479f353a4a0dbcafb99" Nov 01 15:43:30.058498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.058464 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:30.272909 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.272868 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" podUID="6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" containerName="payload-pre-processing" containerID="cri-o://768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d" gracePeriod=25 Nov 01 15:43:30.526977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.526946 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 15:43:30.616394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.616362 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume\") pod \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " Nov 01 15:43:30.616574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.616517 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz\" (UniqueName: \"kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz\") pod \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\" (UID: \"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec\") " Nov 01 15:43:30.616747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.616725 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume" pod "6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" (UID: "6c8d391e-8cd5-4057-a8c5-6570fb3b10ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:30.618532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.618508 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz" pod "6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" (UID: "6c8d391e-8cd5-4057-a8c5-6570fb3b10ec"). InnerVolumeSpecName "kube-api-access-pjzwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:30.630204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.630181 5473 generic.go:356] "Generic (PLEG): container finished" podID="6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" containerID="768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d" exitCode=0 Nov 01 15:43:30.630439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.630249 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" Nov 01 15:43:30.630439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.630248 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" event={"ID":"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec","Type":"ContainerDied","Data":"768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d"} Nov 01 15:43:30.630439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.630364 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442" event={"ID":"6c8d391e-8cd5-4057-a8c5-6570fb3b10ec","Type":"ContainerDied","Data":"e38ad1ab7f16bdd84a15ae116270d43bed7da8236195a57b4f6d92d7b09b1e51"} Nov 01 15:43:30.630439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.630387 5473 scope.go:122] "RemoveContainer" containerID="768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d" Nov 01 15:43:30.640125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.640087 5473 scope.go:122] "RemoveContainer" containerID="768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d" Nov 01 15:43:30.640423 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:30.640402 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d\": container with ID starting with 768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d not found: ID does not exist" containerID="768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d" Nov 01 15:43:30.640477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.640430 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d"} err="failed to get container status \"768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d\": rpc error: code = NotFound desc = could not find container \"768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d\": container with ID starting with 768653de606b4cb93cba5c608360176dd0b220620ad231118d33c137c90a7b7d not found: ID does not exist" Nov 01 15:43:30.717388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.717350 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzwz\" (UniqueName: \"kubernetes.io/projected/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-kube-api-access-pjzwz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:30.717388 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.717383 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:30.752938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.752908 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442"] Nov 01 15:43:30.755588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:30.755562 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5b0dcc-8498f89b7-r8442"] Nov 01 15:43:31.080723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.080691 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" path="/var/lib/kubelet/pods/6c8d391e-8cd5-4057-a8c5-6570fb3b10ec/volumes" Nov 01 15:43:31.109169 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:31.109135 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:43:31.504500 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.504407 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" containerID="cri-o://e66503467f0a237b0f7cb1833e3793aecb28c9c5613de2b90f01a7fc1b5c61a2" gracePeriod=45 Nov 01 15:43:31.639841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.639803 5473 generic.go:356] "Generic (PLEG): container finished" podID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerID="e66503467f0a237b0f7cb1833e3793aecb28c9c5613de2b90f01a7fc1b5c61a2" exitCode=0 Nov 01 15:43:31.640033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.639876 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" event={"ID":"f935c7db-e3c2-49f2-9329-96b7b48b02d3","Type":"ContainerDied","Data":"e66503467f0a237b0f7cb1833e3793aecb28c9c5613de2b90f01a7fc1b5c61a2"} Nov 01 15:43:31.685487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.685452 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf"] Nov 01 15:43:31.774773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.774747 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 15:43:31.791139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.791105 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j"] Nov 01 15:43:31.828458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828428 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir\") pod \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " Nov 01 15:43:31.828627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828492 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache\") pod \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " Nov 01 15:43:31.828627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828570 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs\") pod \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " Nov 01 15:43:31.828771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828642 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht\" (UniqueName: \"kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht\") pod \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " Nov 01 15:43:31.828771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828691 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home\") pod \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " Nov 01 15:43:31.828771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828734 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir" pod "f935c7db-e3c2-49f2-9329-96b7b48b02d3" (UID: "f935c7db-e3c2-49f2-9329-96b7b48b02d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:31.828986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828773 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache" pod "f935c7db-e3c2-49f2-9329-96b7b48b02d3" (UID: "f935c7db-e3c2-49f2-9329-96b7b48b02d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:31.828986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828807 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm\") pod \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\" (UID: \"f935c7db-e3c2-49f2-9329-96b7b48b02d3\") " Nov 01 15:43:31.828986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.828952 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home" pod "f935c7db-e3c2-49f2-9329-96b7b48b02d3" (UID: "f935c7db-e3c2-49f2-9329-96b7b48b02d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:31.829172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.829151 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:31.829172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.829171 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:31.829337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.829180 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:31.830836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.830814 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm" pod "f935c7db-e3c2-49f2-9329-96b7b48b02d3" (UID: "f935c7db-e3c2-49f2-9329-96b7b48b02d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:31.830951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.830893 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs" pod "f935c7db-e3c2-49f2-9329-96b7b48b02d3" (UID: "f935c7db-e3c2-49f2-9329-96b7b48b02d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:43:31.830951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.830896 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht" pod "f935c7db-e3c2-49f2-9329-96b7b48b02d3" (UID: "f935c7db-e3c2-49f2-9329-96b7b48b02d3"). InnerVolumeSpecName "kube-api-access-rb6ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:31.930260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.930227 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f935c7db-e3c2-49f2-9329-96b7b48b02d3-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:31.930260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.930262 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6ht\" (UniqueName: \"kubernetes.io/projected/f935c7db-e3c2-49f2-9329-96b7b48b02d3-kube-api-access-rb6ht\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:31.930476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:31.930275 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f935c7db-e3c2-49f2-9329-96b7b48b02d3-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:32.092755 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.092719 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:32.215987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.215953 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt"] Nov 01 15:43:32.216392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216377 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f45353db-a197-4986-875c-2f4e39564e28" containerName="revoke-keys" Nov 01 15:43:32.216447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216393 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45353db-a197-4986-875c-2f4e39564e28" containerName="revoke-keys" Nov 01 15:43:32.216447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216417 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" Nov 01 15:43:32.216447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216423 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" Nov 01 15:43:32.216447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216435 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" containerName="payload-pre-processing" Nov 01 15:43:32.216447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216441 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" containerName="payload-pre-processing" Nov 01 15:43:32.216621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216502 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" containerName="main" Nov 01 15:43:32.216621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216513 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8d391e-8cd5-4057-a8c5-6570fb3b10ec" containerName="payload-pre-processing" Nov 01 15:43:32.216621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.216523 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45353db-a197-4986-875c-2f4e39564e28" containerName="revoke-keys" Nov 01 15:43:32.219847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.219830 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.261375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.261344 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:32.275703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.275678 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:43:32.275703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.275685 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jt86s\"" Nov 01 15:43:32.279886 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.279862 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt"] Nov 01 15:43:32.334161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.334133 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.334332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.334172 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5z8\" (UniqueName: \"kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.334332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.334283 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.435123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.435052 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.435284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.435150 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.435284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.435191 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5z8\" (UniqueName: \"kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.435804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.435769 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.437494 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.437473 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.491734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.491706 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5z8\" (UniqueName: \"kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8\") pod \"maas-api-revoke-sub-537770ad-pk8gt\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.533107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.533080 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:32.648682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.648657 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" Nov 01 15:43:32.648832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.648702 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh" event={"ID":"f935c7db-e3c2-49f2-9329-96b7b48b02d3","Type":"ContainerDied","Data":"c2749370441bfb8cb07b6eb9bbf8a7591e54c62bfbacb45d8c29dae686381da3"} Nov 01 15:43:32.648832 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.648751 5473 scope.go:122] "RemoveContainer" containerID="e66503467f0a237b0f7cb1833e3793aecb28c9c5613de2b90f01a7fc1b5c61a2" Nov 01 15:43:32.716924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.716867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:32.812820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.812792 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh"] Nov 01 15:43:32.816161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.816137 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5b0dcc/simulator-w3-5b0dcc-kserve-5797986977-gtbbh"] Nov 01 15:43:32.913285 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:43:32.913256 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd908adfd_9c4b_4bcc_a702_7e466d7fcf94.slice/crio-de556d8144dc54904cede5a9a44ff3580de984f8e50ff92f04af50a8aa0ed538 WatchSource:0}: Error finding container de556d8144dc54904cede5a9a44ff3580de984f8e50ff92f04af50a8aa0ed538: Status 404 returned error can't find the container with id de556d8144dc54904cede5a9a44ff3580de984f8e50ff92f04af50a8aa0ed538 Nov 01 15:43:32.917087 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:32.917066 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt"] Nov 01 15:43:33.080750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:33.080721 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f935c7db-e3c2-49f2-9329-96b7b48b02d3" path="/var/lib/kubelet/pods/f935c7db-e3c2-49f2-9329-96b7b48b02d3/volumes" Nov 01 15:43:33.655737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:33.655701 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" event={"ID":"d908adfd-9c4b-4bcc-a702-7e466d7fcf94","Type":"ContainerStarted","Data":"53cdb2316545a0bc3b06c9acd986db99a017f9109327a0151c891ebcb90b0699"} Nov 01 15:43:33.656279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:33.655788 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" event={"ID":"d908adfd-9c4b-4bcc-a702-7e466d7fcf94","Type":"ContainerStarted","Data":"de556d8144dc54904cede5a9a44ff3580de984f8e50ff92f04af50a8aa0ed538"} Nov 01 15:43:33.769050 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:33.768976 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" podStartSLOduration=1.768957846 podStartE2EDuration="1.768957846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:43:33.768254963 +0000 UTC m=+5152.826406819" watchObservedRunningTime="2026-11-01 15:43:33.768957846 +0000 UTC m=+5152.827109696" Nov 01 15:43:34.664117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:34.664074 5473 generic.go:356] "Generic (PLEG): container finished" podID="d908adfd-9c4b-4bcc-a702-7e466d7fcf94" containerID="53cdb2316545a0bc3b06c9acd986db99a017f9109327a0151c891ebcb90b0699" exitCode=0 Nov 01 15:43:34.664569 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:34.664157 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" event={"ID":"d908adfd-9c4b-4bcc-a702-7e466d7fcf94","Type":"ContainerDied","Data":"53cdb2316545a0bc3b06c9acd986db99a017f9109327a0151c891ebcb90b0699"} Nov 01 15:43:35.032549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.032510 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79"] Nov 01 15:43:35.036439 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.036418 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.091617 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.091583 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 15:43:35.091807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.091748 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5b0dcc-openshift-default-dockercfg-4d7r2\"" Nov 01 15:43:35.095640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.095612 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79"] Nov 01 15:43:35.158313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158275 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158340 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158425 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158487 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158524 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158561 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxf87\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158676 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158695 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.158776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.158714 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260001 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260043 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260062 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260104 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260120 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260146 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260198 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260265 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260323 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxf87\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260441 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260546 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260583 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.260734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.260681 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.261076 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.261057 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.262562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.262541 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.262794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.262776 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.318359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.318283 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.318505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.318402 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxf87\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87\") pod \"e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.349765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.349736 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:35.764797 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:43:35.764728 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3018a37_3dbc_4f2d_b066_7750b13e6234.slice/crio-a340b9160c80cf41805772b8f209a49a4221e4c30cb6daece2244e37ca7f8a8a WatchSource:0}: Error finding container a340b9160c80cf41805772b8f209a49a4221e4c30cb6daece2244e37ca7f8a8a: Status 404 returned error can't find the container with id a340b9160c80cf41805772b8f209a49a4221e4c30cb6daece2244e37ca7f8a8a Nov 01 15:43:35.767510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.767469 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:43:35.767621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.767557 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:43:35.767621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.767598 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:43:35.768987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.768965 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79"] Nov 01 15:43:35.791391 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.791372 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:35.828511 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.828489 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:35.867025 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.866993 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8\" (UniqueName: \"kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8\") pod \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " Nov 01 15:43:35.867198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.867072 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca\") pod \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " Nov 01 15:43:35.867198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.867106 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca\") pod \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\" (UID: \"d908adfd-9c4b-4bcc-a702-7e466d7fcf94\") " Nov 01 15:43:35.867876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.867848 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca" pod "d908adfd-9c4b-4bcc-a702-7e466d7fcf94" (UID: "d908adfd-9c4b-4bcc-a702-7e466d7fcf94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:35.869383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.869360 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8" pod "d908adfd-9c4b-4bcc-a702-7e466d7fcf94" (UID: "d908adfd-9c4b-4bcc-a702-7e466d7fcf94"). InnerVolumeSpecName "kube-api-access-vg5z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:35.869827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.869801 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca" pod "d908adfd-9c4b-4bcc-a702-7e466d7fcf94" (UID: "d908adfd-9c4b-4bcc-a702-7e466d7fcf94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:43:35.968498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.968417 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5z8\" (UniqueName: \"kubernetes.io/projected/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-kube-api-access-vg5z8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:35.968498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.968444 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:35.968498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:35.968454 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d908adfd-9c4b-4bcc-a702-7e466d7fcf94-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:36.677529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.677423 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" event={"ID":"d3018a37-3dbc-4f2d-b066-7750b13e6234","Type":"ContainerStarted","Data":"2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b"} Nov 01 15:43:36.677529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.677486 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" event={"ID":"d3018a37-3dbc-4f2d-b066-7750b13e6234","Type":"ContainerStarted","Data":"a340b9160c80cf41805772b8f209a49a4221e4c30cb6daece2244e37ca7f8a8a"} Nov 01 15:43:36.678786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.678763 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" Nov 01 15:43:36.678908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.678799 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt" event={"ID":"d908adfd-9c4b-4bcc-a702-7e466d7fcf94","Type":"ContainerDied","Data":"de556d8144dc54904cede5a9a44ff3580de984f8e50ff92f04af50a8aa0ed538"} Nov 01 15:43:36.678908 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.678838 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de556d8144dc54904cede5a9a44ff3580de984f8e50ff92f04af50a8aa0ed538" Nov 01 15:43:36.686063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.686025 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" podUID="521f5798-1315-4865-ab33-a8314f7b6e15" containerName="payload-processing" containerID="cri-o://684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d" gracePeriod=25 Nov 01 15:43:36.792173 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.792136 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" podUID="8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" containerName="payload-pre-processing" containerID="cri-o://ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779" gracePeriod=25 Nov 01 15:43:36.805164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.805109 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podStartSLOduration=2.805095957 podStartE2EDuration="2.805095957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:43:36.803570329 +0000 UTC m=+5155.861722178" watchObservedRunningTime="2026-11-01 15:43:36.805095957 +0000 UTC m=+5155.863247806" Nov 01 15:43:36.979537 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:36.979506 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 15:43:37.062740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.062713 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 15:43:37.080434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.080404 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5\" (UniqueName: \"kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5\") pod \"521f5798-1315-4865-ab33-a8314f7b6e15\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " Nov 01 15:43:37.080553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.080460 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume\") pod \"521f5798-1315-4865-ab33-a8314f7b6e15\" (UID: \"521f5798-1315-4865-ab33-a8314f7b6e15\") " Nov 01 15:43:37.080799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.080775 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume" pod "521f5798-1315-4865-ab33-a8314f7b6e15" (UID: "521f5798-1315-4865-ab33-a8314f7b6e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:37.080872 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.080851 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521f5798-1315-4865-ab33-a8314f7b6e15-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:37.082512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.082490 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5" pod "521f5798-1315-4865-ab33-a8314f7b6e15" (UID: "521f5798-1315-4865-ab33-a8314f7b6e15"). InnerVolumeSpecName "kube-api-access-qc7h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:37.181771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.181739 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7\" (UniqueName: \"kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7\") pod \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " Nov 01 15:43:37.181976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.181784 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume\") pod \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\" (UID: \"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511\") " Nov 01 15:43:37.182417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.182189 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7h5\" (UniqueName: \"kubernetes.io/projected/521f5798-1315-4865-ab33-a8314f7b6e15-kube-api-access-qc7h5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:37.182417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.182236 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume" pod "8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" (UID: "8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:43:37.183958 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.183933 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7" pod "8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" (UID: "8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511"). InnerVolumeSpecName "kube-api-access-v9qz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:37.283011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.282982 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9qz7\" (UniqueName: \"kubernetes.io/projected/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-kube-api-access-v9qz7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:37.283011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.283011 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:37.298754 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.298727 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:37.350423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.350389 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:37.352290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.352264 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body= Nov 01 15:43:37.352417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.352311 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" Nov 01 15:43:37.685976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.685896 5473 generic.go:356] "Generic (PLEG): container finished" podID="521f5798-1315-4865-ab33-a8314f7b6e15" containerID="684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d" exitCode=0 Nov 01 15:43:37.685976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.685941 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" Nov 01 15:43:37.686259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.685978 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" event={"ID":"521f5798-1315-4865-ab33-a8314f7b6e15","Type":"ContainerDied","Data":"684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d"} Nov 01 15:43:37.686259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.686037 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf" event={"ID":"521f5798-1315-4865-ab33-a8314f7b6e15","Type":"ContainerDied","Data":"ef4f457ec953cfbfcb65f021c34d2275be6d238186666873b244095803929143"} Nov 01 15:43:37.686259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.686067 5473 scope.go:122] "RemoveContainer" containerID="684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d" Nov 01 15:43:37.687584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.687557 5473 generic.go:356] "Generic (PLEG): container finished" podID="8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" containerID="ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779" exitCode=0 Nov 01 15:43:37.687744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.687606 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" Nov 01 15:43:37.687744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.687641 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" event={"ID":"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511","Type":"ContainerDied","Data":"ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779"} Nov 01 15:43:37.687744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.687703 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j" event={"ID":"8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511","Type":"ContainerDied","Data":"d411c562ea56afad99acc252153db54ff60a4fbe1a6bdf6296ffcf04184ddeed"} Nov 01 15:43:37.696174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.696152 5473 scope.go:122] "RemoveContainer" containerID="684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d" Nov 01 15:43:37.696488 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:37.696469 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d\": container with ID starting with 684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d not found: ID does not exist" containerID="684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d" Nov 01 15:43:37.696565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.696496 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d"} err="failed to get container status \"684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d\": rpc error: code = NotFound desc = could not find container \"684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d\": container with ID starting with 684f48003c2d64dbaf5b384937b08b3929dbe4588e5eb3d0eadb94329505cd8d not found: ID does not exist" Nov 01 15:43:37.696565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.696512 5473 scope.go:122] "RemoveContainer" containerID="ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779" Nov 01 15:43:37.704932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.704916 5473 scope.go:122] "RemoveContainer" containerID="ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779" Nov 01 15:43:37.705141 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:43:37.705122 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779\": container with ID starting with ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779 not found: ID does not exist" containerID="ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779" Nov 01 15:43:37.705192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.705147 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779"} err="failed to get container status \"ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779\": rpc error: code = NotFound desc = could not find container \"ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779\": container with ID starting with ebf3944db8459bb097d0f13e6962385183e57d3eb9a25a1e322e01a682e5e779 not found: ID does not exist" Nov 01 15:43:37.855011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.854970 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j"] Nov 01 15:43:37.858917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.858894 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-00a787-66858fdf87-w5w2j"] Nov 01 15:43:37.969415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.969118 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf"] Nov 01 15:43:37.975318 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:37.975283 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-00a787-976fd8987-7dncf"] Nov 01 15:43:38.350695 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:38.350653 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body= Nov 01 15:43:38.350876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:38.350722 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" Nov 01 15:43:38.482895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:38.482859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:39.080091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.080057 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521f5798-1315-4865-ab33-a8314f7b6e15" path="/var/lib/kubelet/pods/521f5798-1315-4865-ab33-a8314f7b6e15/volumes" Nov 01 15:43:39.080485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.080471 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" path="/var/lib/kubelet/pods/8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511/volumes" Nov 01 15:43:39.350711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.350629 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body= Nov 01 15:43:39.350711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.350682 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" Nov 01 15:43:39.451148 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.451112 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" containerID="cri-o://9a2b0a70ffa17a078add9ab153c7705186ed51127e3a670961703b2e9f157670" gracePeriod=45 Nov 01 15:43:39.704599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.704569 5473 generic.go:356] "Generic (PLEG): container finished" podID="47537040-036d-45ff-88fa-e059edbaa5aa" containerID="9a2b0a70ffa17a078add9ab153c7705186ed51127e3a670961703b2e9f157670" exitCode=0 Nov 01 15:43:39.704744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.704636 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" event={"ID":"47537040-036d-45ff-88fa-e059edbaa5aa","Type":"ContainerDied","Data":"9a2b0a70ffa17a078add9ab153c7705186ed51127e3a670961703b2e9f157670"} Nov 01 15:43:39.704744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.704675 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" event={"ID":"47537040-036d-45ff-88fa-e059edbaa5aa","Type":"ContainerDied","Data":"ac72d88269dce2d17876636acd2f0a818513b8793a5e59fe1b67d9c84cd39376"} Nov 01 15:43:39.704744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.704688 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac72d88269dce2d17876636acd2f0a818513b8793a5e59fe1b67d9c84cd39376" Nov 01 15:43:39.705363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.705336 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 15:43:39.806088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806058 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h\" (UniqueName: \"kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h\") pod \"47537040-036d-45ff-88fa-e059edbaa5aa\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " Nov 01 15:43:39.806293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806110 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home\") pod \"47537040-036d-45ff-88fa-e059edbaa5aa\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " Nov 01 15:43:39.806293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806138 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs\") pod \"47537040-036d-45ff-88fa-e059edbaa5aa\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " Nov 01 15:43:39.806293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806271 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm\") pod \"47537040-036d-45ff-88fa-e059edbaa5aa\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " Nov 01 15:43:39.806483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806343 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir\") pod \"47537040-036d-45ff-88fa-e059edbaa5aa\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " Nov 01 15:43:39.806483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806405 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache\") pod \"47537040-036d-45ff-88fa-e059edbaa5aa\" (UID: \"47537040-036d-45ff-88fa-e059edbaa5aa\") " Nov 01 15:43:39.806483 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806452 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home" pod "47537040-036d-45ff-88fa-e059edbaa5aa" (UID: "47537040-036d-45ff-88fa-e059edbaa5aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:39.806657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806610 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir" pod "47537040-036d-45ff-88fa-e059edbaa5aa" (UID: "47537040-036d-45ff-88fa-e059edbaa5aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:39.806720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806671 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:39.806773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.806748 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache" pod "47537040-036d-45ff-88fa-e059edbaa5aa" (UID: "47537040-036d-45ff-88fa-e059edbaa5aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:39.808455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.808431 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm" pod "47537040-036d-45ff-88fa-e059edbaa5aa" (UID: "47537040-036d-45ff-88fa-e059edbaa5aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:43:39.808550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.808484 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs" pod "47537040-036d-45ff-88fa-e059edbaa5aa" (UID: "47537040-036d-45ff-88fa-e059edbaa5aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:43:39.808550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.808515 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h" pod "47537040-036d-45ff-88fa-e059edbaa5aa" (UID: "47537040-036d-45ff-88fa-e059edbaa5aa"). InnerVolumeSpecName "kube-api-access-72j6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:43:39.907466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.907389 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47537040-036d-45ff-88fa-e059edbaa5aa-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:39.907466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.907414 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:39.907466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.907424 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:39.907466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.907434 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47537040-036d-45ff-88fa-e059edbaa5aa-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:39.907466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:39.907444 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72j6h\" (UniqueName: \"kubernetes.io/projected/47537040-036d-45ff-88fa-e059edbaa5aa-kube-api-access-72j6h\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:43:40.350621 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:40.350583 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body= Nov 01 15:43:40.351004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:40.350645 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" Nov 01 15:43:40.710392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:40.710313 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg" Nov 01 15:43:40.873801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:40.873765 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg"] Nov 01 15:43:40.876567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:40.876545 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-00a787/premium-w1-00a787-kserve-54c6fc6f86-rvclg"] Nov 01 15:43:41.081059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:41.081019 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" path="/var/lib/kubelet/pods/47537040-036d-45ff-88fa-e059edbaa5aa/volumes" Nov 01 15:43:41.350763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:41.350673 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body= Nov 01 15:43:41.350763 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:41.350739 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" Nov 01 15:43:41.456036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:41.455998 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:41.623596 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:41.623516 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:42.351017 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:42.350986 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" start-of-body= Nov 01 15:43:42.351454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:42.351038 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.99:15021/healthz/ready\": dial tcp 10.134.1.99:15021: connect: connection refused" Nov 01 15:43:42.813319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:42.813286 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:44.058999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:44.058966 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:44.059481 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:44.059268 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:44.060155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:44.060130 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:43:44.517739 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:44.517715 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:44.712006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:44.711973 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:44.960558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:44.960468 5473 scope.go:122] "RemoveContainer" containerID="9a2b0a70ffa17a078add9ab153c7705186ed51127e3a670961703b2e9f157670" Nov 01 15:43:50.048163 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048116 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l"] Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048507 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" containerName="payload-pre-processing" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048520 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" containerName="payload-pre-processing" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048536 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d908adfd-9c4b-4bcc-a702-7e466d7fcf94" containerName="revoke-keys" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048541 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d908adfd-9c4b-4bcc-a702-7e466d7fcf94" containerName="revoke-keys" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048556 5473 cpu_manager.go:414] "containerMap: removing container" podUID="521f5798-1315-4865-ab33-a8314f7b6e15" containerName="payload-processing" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048561 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521f5798-1315-4865-ab33-a8314f7b6e15" containerName="payload-processing" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048577 5473 cpu_manager.go:414] "containerMap: removing container" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048582 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048639 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="47537040-036d-45ff-88fa-e059edbaa5aa" containerName="main" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048651 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfb3f3b-fdcb-4bb5-9158-69c7b1d94511" containerName="payload-pre-processing" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048660 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d908adfd-9c4b-4bcc-a702-7e466d7fcf94" containerName="revoke-keys" Nov 01 15:43:50.048667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.048668 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="521f5798-1315-4865-ab33-a8314f7b6e15" containerName="payload-processing" Nov 01 15:43:50.053424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.053405 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.105896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.105870 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-00a787-openshift-default-dockercfg-v7d89\"" Nov 01 15:43:50.160639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.160610 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l"] Nov 01 15:43:50.220657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.220630 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.220798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.220664 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.220798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.220685 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.220798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.220746 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.220955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.220883 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.220993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.220964 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.221029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.221002 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs5p\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.221065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.221032 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.221115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.221093 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322393 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322433 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322468 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322455 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322566 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322611 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322647 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322673 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs5p\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322695 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.322743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322718 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.323029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322857 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.323029 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.322893 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.323107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.323064 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.323454 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.323432 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.323550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.323429 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.325019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.324994 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.325115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.324996 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.379114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.379086 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.379228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.379179 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs5p\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p\") pod \"e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:50.666267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:50.666170 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:51.089932 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:43:51.089901 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36e2ac9_08cf_4856_9eef_980bbe570676.slice/crio-5a23fbbfa4582a414c6a0bc4612f528fb5ae94177eef88083699bcc1d37739de WatchSource:0}: Error finding container 5a23fbbfa4582a414c6a0bc4612f528fb5ae94177eef88083699bcc1d37739de: Status 404 returned error can't find the container with id 5a23fbbfa4582a414c6a0bc4612f528fb5ae94177eef88083699bcc1d37739de Nov 01 15:43:51.090288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:51.090188 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l"] Nov 01 15:43:51.091854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:51.091826 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:43:51.091925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:51.091883 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:43:51.091925 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:51.091909 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:43:51.778956 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:51.778914 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" event={"ID":"b36e2ac9-08cf-4856-9eef-980bbe570676","Type":"ContainerStarted","Data":"b8f78809ade864e603a805f042f8541c4aa87f5fa7c1d16368d39668b309287b"} Nov 01 15:43:51.779176 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:51.778983 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" event={"ID":"b36e2ac9-08cf-4856-9eef-980bbe570676","Type":"ContainerStarted","Data":"5a23fbbfa4582a414c6a0bc4612f528fb5ae94177eef88083699bcc1d37739de"} Nov 01 15:43:52.007317 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:52.007252 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podStartSLOduration=2.007231371 podStartE2EDuration="2.007231371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:43:51.955942098 +0000 UTC m=+5171.014093946" watchObservedRunningTime="2026-11-01 15:43:52.007231371 +0000 UTC m=+5171.065383219" Nov 01 15:43:52.666299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:52.666264 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:52.667700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:52.667674 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body= Nov 01 15:43:52.667842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:52.667732 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" Nov 01 15:43:53.113272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:53.113243 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:53.288410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:53.288369 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:53.666884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:53.666849 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body= Nov 01 15:43:53.667330 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:53.666899 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" Nov 01 15:43:54.243239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:54.239647 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:54.667489 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:54.667450 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body= Nov 01 15:43:54.667898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:54.667510 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" Nov 01 15:43:55.667409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:55.667374 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body= Nov 01 15:43:55.667604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:55.667431 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" Nov 01 15:43:55.878411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:55.878384 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:56.310699 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.310667 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79"] Nov 01 15:43:56.310885 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.310863 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" containerID="cri-o://2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b" gracePeriod=30 Nov 01 15:43:56.313906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.313817 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:56.666918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.666838 5473 patch_prober.go:29] interesting pod/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" start-of-body= Nov 01 15:43:56.666918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.666894 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.100:15021/healthz/ready\": dial tcp 10.134.1.100:15021: connect: connection refused" Nov 01 15:43:56.812811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.812760 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:56.970670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:56.970593 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:43:57.973814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:57.973779 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:57.974336 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:57.974321 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:57.975253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:57.975207 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:43:59.061457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:59.061419 5473 patch_prober.go:29] interesting pod/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:43:59.061852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:59.061483 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:43:59.912762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:43:59.912726 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:00.076651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:00.076618 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:00.953289 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:00.953259 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:01.106855 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:44:01.106815 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:44:01.114523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.114498 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:01.561682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.561660 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:44:01.642588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642516 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642563 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642588 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642622 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642646 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642677 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642756 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642796 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642812 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.642839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642815 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:01.643268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.642855 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert\") pod \"d3018a37-3dbc-4f2d-b066-7750b13e6234\" (UID: \"d3018a37-3dbc-4f2d-b066-7750b13e6234\") " Nov 01 15:44:01.643268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643060 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:01.643268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643114 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:01.643268 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643259 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.643512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643281 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.643512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643296 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.643512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643311 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.643512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.643387 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:44:01.644850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.644825 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:01.644989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.644962 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:44:01.645041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.644977 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "kube-api-access-qxf87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:44:01.645041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.644996 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token" pod "d3018a37-3dbc-4f2d-b066-7750b13e6234" (UID: "d3018a37-3dbc-4f2d-b066-7750b13e6234"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:44:01.744783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.744738 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3018a37-3dbc-4f2d-b066-7750b13e6234-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.744783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.744776 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxf87\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-kube-api-access-qxf87\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.744783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.744787 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.744783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.744796 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.745052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.744805 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3018a37-3dbc-4f2d-b066-7750b13e6234-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:01.838398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.838366 5473 generic.go:356] "Generic (PLEG): container finished" podID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerID="2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b" exitCode=0 Nov 01 15:44:01.838550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.838414 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" Nov 01 15:44:01.838550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.838454 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" event={"ID":"d3018a37-3dbc-4f2d-b066-7750b13e6234","Type":"ContainerDied","Data":"2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b"} Nov 01 15:44:01.838550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.838510 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79" event={"ID":"d3018a37-3dbc-4f2d-b066-7750b13e6234","Type":"ContainerDied","Data":"a340b9160c80cf41805772b8f209a49a4221e4c30cb6daece2244e37ca7f8a8a"} Nov 01 15:44:01.838550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.838535 5473 scope.go:122] "RemoveContainer" containerID="2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b" Nov 01 15:44:01.848982 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.848967 5473 scope.go:122] "RemoveContainer" containerID="2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b" Nov 01 15:44:01.849252 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:44:01.849228 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b\": container with ID starting with 2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b not found: ID does not exist" containerID="2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b" Nov 01 15:44:01.849326 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.849265 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b"} err="failed to get container status \"2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b\": rpc error: code = NotFound desc = could not find container \"2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b\": container with ID starting with 2865fa68901fa31969613b5412d994ce4055163e016f5bdd0258ab61b479a92b not found: ID does not exist" Nov 01 15:44:01.959311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:01.959193 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:02.003689 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:02.003654 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79"] Nov 01 15:44:02.007912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:02.007888 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5b0dcc-openshift-default-8568746c89-gfx79"] Nov 01 15:44:02.053303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:02.053276 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp"] Nov 01 15:44:02.057429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:02.057404 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-029de10c-db9qp"] Nov 01 15:44:03.080041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:03.080009 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" path="/var/lib/kubelet/pods/d3018a37-3dbc-4f2d-b066-7750b13e6234/volumes" Nov 01 15:44:03.080449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:03.080433 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff71f8d-a340-4686-b832-236fc4b2728f" path="/var/lib/kubelet/pods/eff71f8d-a340-4686-b832-236fc4b2728f/volumes" Nov 01 15:44:03.256938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:03.256907 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:03.694078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:03.694041 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l"] Nov 01 15:44:03.694308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:03.694282 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" containerID="cri-o://b8f78809ade864e603a805f042f8541c4aa87f5fa7c1d16368d39668b309287b" gracePeriod=30 Nov 01 15:44:03.718476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:03.718448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:07.100944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:07.100912 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6gn9k_2a92488d-7096-4b4c-ab4f-79bf4dd16e94/manager/0.log" Nov 01 15:44:07.493061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:07.492964 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:08.884109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:08.884080 5473 generic.go:356] "Generic (PLEG): container finished" podID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerID="b8f78809ade864e603a805f042f8541c4aa87f5fa7c1d16368d39668b309287b" exitCode=0 Nov 01 15:44:08.884502 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:08.884149 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" event={"ID":"b36e2ac9-08cf-4856-9eef-980bbe570676","Type":"ContainerDied","Data":"b8f78809ade864e603a805f042f8541c4aa87f5fa7c1d16368d39668b309287b"} Nov 01 15:44:08.942514 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:08.942492 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:44:09.005705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005649 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.005835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005711 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.005835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005731 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.005835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005751 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.005835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005782 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.005835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005810 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.005835 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005834 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.006132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005868 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.006132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005910 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p\") pod \"b36e2ac9-08cf-4856-9eef-980bbe570676\" (UID: \"b36e2ac9-08cf-4856-9eef-980bbe570676\") " Nov 01 15:44:09.006132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.005932 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:09.006132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.006015 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:09.006410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.006191 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.006410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.006254 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:09.006410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.006274 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.006410 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.006384 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:44:09.006635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.006439 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:09.008166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.008144 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:44:09.008350 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.008181 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:44:09.008546 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.008517 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:44:09.008686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.008668 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p" pod "b36e2ac9-08cf-4856-9eef-980bbe570676" (UID: "b36e2ac9-08cf-4856-9eef-980bbe570676"). InnerVolumeSpecName "kube-api-access-nqs5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:44:09.107285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107259 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.107444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107311 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.107444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107321 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.107444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107330 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36e2ac9-08cf-4856-9eef-980bbe570676-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.107444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107339 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.107444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107347 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36e2ac9-08cf-4856-9eef-980bbe570676-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.107444 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.107356 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqs5p\" (UniqueName: \"kubernetes.io/projected/b36e2ac9-08cf-4856-9eef-980bbe570676-kube-api-access-nqs5p\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:09.890849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.890823 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" Nov 01 15:44:09.890849 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.890833 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l" event={"ID":"b36e2ac9-08cf-4856-9eef-980bbe570676","Type":"ContainerDied","Data":"5a23fbbfa4582a414c6a0bc4612f528fb5ae94177eef88083699bcc1d37739de"} Nov 01 15:44:09.891376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:09.890920 5473 scope.go:122] "RemoveContainer" containerID="b8f78809ade864e603a805f042f8541c4aa87f5fa7c1d16368d39668b309287b" Nov 01 15:44:10.056424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:10.056394 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l"] Nov 01 15:44:10.106608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:10.106580 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-00a787-openshift-default-6d97564689-pfj5l"] Nov 01 15:44:10.214807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:10.214738 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gwmh8_2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7/manager/0.log" Nov 01 15:44:10.612090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:10.612057 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:11.079891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:11.079859 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" path="/var/lib/kubelet/pods/b36e2ac9-08cf-4856-9eef-980bbe570676/volumes" Nov 01 15:44:13.046944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:13.046914 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz"] Nov 01 15:44:13.100962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:13.100933 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-543f285d-k54lz"] Nov 01 15:44:13.368211 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:13.368127 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:15.080303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.080265 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff" path="/var/lib/kubelet/pods/7bff4d90-0ffa-4a57-bc9c-38d5e8ad53ff/volumes" Nov 01 15:44:15.290290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.290256 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:15.761119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761084 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr"] Nov 01 15:44:15.761503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761488 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" Nov 01 15:44:15.761555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761504 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" Nov 01 15:44:15.761555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761523 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" Nov 01 15:44:15.761555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761530 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" Nov 01 15:44:15.761656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761619 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36e2ac9-08cf-4856-9eef-980bbe570676" containerName="istio-proxy" Nov 01 15:44:15.761656 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.761627 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3018a37-3dbc-4f2d-b066-7750b13e6234" containerName="istio-proxy" Nov 01 15:44:15.766064 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.766048 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.818806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.818772 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 15:44:15.832436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.832416 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1489d8ef-openshift-default-dockercfg-7xmmg\"" Nov 01 15:44:15.869275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869236 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lph68\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869311 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869332 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869392 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869411 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869472 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869487 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869544 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.869683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.869566 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.873462 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.873436 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr"] Nov 01 15:44:15.920810 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.920782 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:15.970461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970434 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970465 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970518 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970535 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970560 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970583 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970621 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lph68\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970656 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.970896 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970681 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.971068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.970996 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.971118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.971059 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.971118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.971108 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.971202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.971156 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.971394 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.971368 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.973146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.973120 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:15.973259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:15.973244 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:16.027829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.027796 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:16.028014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.027995 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lph68\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68\") pod \"e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:16.082418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.082391 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:16.127666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.127627 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:16.496264 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:44:16.496208 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389eaa90_162b_477b_876c_09cf48f97de8.slice/crio-4cadcec85907baa44b57ed65f38b4bd5dad9c71937dd438a6d889bc427b98546 WatchSource:0}: Error finding container 4cadcec85907baa44b57ed65f38b4bd5dad9c71937dd438a6d889bc427b98546: Status 404 returned error can't find the container with id 4cadcec85907baa44b57ed65f38b4bd5dad9c71937dd438a6d889bc427b98546 Nov 01 15:44:16.502338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.502259 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr"] Nov 01 15:44:16.506781 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.506743 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:44:16.506847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.506820 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:44:16.506901 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.506859 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:44:16.935023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.934988 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" event={"ID":"389eaa90-162b-477b-876c-09cf48f97de8","Type":"ContainerStarted","Data":"b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36"} Nov 01 15:44:16.935269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:16.935046 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" event={"ID":"389eaa90-162b-477b-876c-09cf48f97de8","Type":"ContainerStarted","Data":"4cadcec85907baa44b57ed65f38b4bd5dad9c71937dd438a6d889bc427b98546"} Nov 01 15:44:17.083377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:17.083346 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:17.088280 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:17.088256 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:17.102559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:17.102507 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podStartSLOduration=2.102496285 podStartE2EDuration="2.102496285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:44:17.101312896 +0000 UTC m=+5196.159464746" watchObservedRunningTime="2026-11-01 15:44:17.102496285 +0000 UTC m=+5196.160648136" Nov 01 15:44:17.941275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:17.941246 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:17.942601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:17.942580 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:18.877577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:18.877541 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:19.040761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:19.040727 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:19.238986 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:19.238909 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:19.470841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:19.470810 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:22.289236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.289192 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:22.453332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.453300 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:22.602650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.602577 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb"] Nov 01 15:44:22.606303 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.606287 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.662878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.662843 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb"] Nov 01 15:44:22.708938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.708898 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1489d8ef-dockercfg-772gq\"" Nov 01 15:44:22.708938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.708907 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1489d8ef\"" Nov 01 15:44:22.729616 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.729588 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.729756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.729628 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh66\" (UniqueName: \"kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66\") pod \"payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.766556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.766518 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh"] Nov 01 15:44:22.770396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.770377 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:22.827558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.827536 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh"] Nov 01 15:44:22.830448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.830422 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.830582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.830455 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:22.830582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.830478 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh66\" (UniqueName: \"kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66\") pod \"payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.830713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.830634 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7xzl\" (UniqueName: \"kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl\") pod \"payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:22.831158 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.831135 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.888397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.888344 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh66\" (UniqueName: \"kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66\") pod \"payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.919321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.919297 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:22.931657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.931628 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:22.931826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.931708 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7xzl\" (UniqueName: \"kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl\") pod \"payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:22.932329 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.932312 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume\") pod \"payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:22.988948 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:22.988921 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7xzl\" (UniqueName: \"kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl\") pod \"payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:23.084113 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.084081 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:23.282070 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:44:23.282032 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19772580_8033_4e11_9b86_70f37d555d47.slice/crio-b4637f2b13aeb8574eca5888e88a331c9542ea881a2a89d2a8dbe5f32e81dc32 WatchSource:0}: Error finding container b4637f2b13aeb8574eca5888e88a331c9542ea881a2a89d2a8dbe5f32e81dc32: Status 404 returned error can't find the container with id b4637f2b13aeb8574eca5888e88a331c9542ea881a2a89d2a8dbe5f32e81dc32 Nov 01 15:44:23.284762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.284736 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb"] Nov 01 15:44:23.449818 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:44:23.449787 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e57fb0e_1743_4cdd_be95_2b5959d875ab.slice/crio-1d2bbbd4f3d91d1f90d41dbb7335dddc231a26a864717ac6d94e4a30dd9f3fe6 WatchSource:0}: Error finding container 1d2bbbd4f3d91d1f90d41dbb7335dddc231a26a864717ac6d94e4a30dd9f3fe6: Status 404 returned error can't find the container with id 1d2bbbd4f3d91d1f90d41dbb7335dddc231a26a864717ac6d94e4a30dd9f3fe6 Nov 01 15:44:23.453705 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.453676 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh"] Nov 01 15:44:23.979166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.979129 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" event={"ID":"4e57fb0e-1743-4cdd-be95-2b5959d875ab","Type":"ContainerStarted","Data":"7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff"} Nov 01 15:44:23.979399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.979182 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" event={"ID":"4e57fb0e-1743-4cdd-be95-2b5959d875ab","Type":"ContainerStarted","Data":"1d2bbbd4f3d91d1f90d41dbb7335dddc231a26a864717ac6d94e4a30dd9f3fe6"} Nov 01 15:44:23.979399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.979233 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:23.980782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.980757 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" event={"ID":"19772580-8033-4e11-9b86-70f37d555d47","Type":"ContainerStarted","Data":"e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd"} Nov 01 15:44:23.980919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.980793 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" event={"ID":"19772580-8033-4e11-9b86-70f37d555d47","Type":"ContainerStarted","Data":"b4637f2b13aeb8574eca5888e88a331c9542ea881a2a89d2a8dbe5f32e81dc32"} Nov 01 15:44:23.980919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:23.980813 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:24.093631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:24.093582 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" podStartSLOduration=2.093568883 podStartE2EDuration="2.093568883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:44:24.091283058 +0000 UTC m=+5203.149434910" watchObservedRunningTime="2026-11-01 15:44:24.093568883 +0000 UTC m=+5203.151720732" Nov 01 15:44:24.208187 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:24.208120 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" podStartSLOduration=2.208101458 podStartE2EDuration="2.208101458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:44:24.205632022 +0000 UTC m=+5203.263783872" watchObservedRunningTime="2026-11-01 15:44:24.208101458 +0000 UTC m=+5203.266253311" Nov 01 15:44:24.853591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:24.853559 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:25.021543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:25.021502 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:25.050183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:25.050151 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw"] Nov 01 15:44:25.099691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:25.099660 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2f6319b-ml6xw"] Nov 01 15:44:25.180643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:25.180563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:27.080662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:27.080629 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f861b9a-c205-402e-ad42-452194b30128" path="/var/lib/kubelet/pods/6f861b9a-c205-402e-ad42-452194b30128/volumes" Nov 01 15:44:29.387333 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:29.387294 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:31.106736 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:44:31.106702 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:44:33.113683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:33.113651 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm"] Nov 01 15:44:33.163372 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:33.163337 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6f39f2-ddfjm"] Nov 01 15:44:33.911921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:33.911888 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:34.989617 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:34.989589 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:44:34.990020 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:34.989654 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:44:35.080877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:35.080842 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2506ab-b32e-4b41-81b3-f3e54dc10bf2" path="/var/lib/kubelet/pods/2e2506ab-b32e-4b41-81b3-f3e54dc10bf2/volumes" Nov 01 15:44:36.051863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:36.051824 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn"] Nov 01 15:44:36.101349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:36.101315 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7b795f7-99hvn"] Nov 01 15:44:37.081105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:37.081072 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c806575-f331-440c-97f7-ec39ef4d975d" path="/var/lib/kubelet/pods/9c806575-f331-440c-97f7-ec39ef4d975d/volumes" Nov 01 15:44:38.292679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:38.292642 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:42.112670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.112640 5473 generic.go:356] "Generic (PLEG): container finished" podID="389eaa90-162b-477b-876c-09cf48f97de8" containerID="b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36" exitCode=0 Nov 01 15:44:42.113111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.112709 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" event={"ID":"389eaa90-162b-477b-876c-09cf48f97de8","Type":"ContainerDied","Data":"b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36"} Nov 01 15:44:42.113111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.113024 5473 scope.go:122] "RemoveContainer" containerID="b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36" Nov 01 15:44:42.114172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.114145 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:44:42.114271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.114254 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:44:42.114324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.114292 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:44:42.982174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:42.982138 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:43.123918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:43.123886 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" event={"ID":"389eaa90-162b-477b-876c-09cf48f97de8","Type":"ContainerStarted","Data":"51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586"} Nov 01 15:44:44.051015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:44.050981 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng"] Nov 01 15:44:44.082819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:44.082791 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:44.083985 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:44.083962 5473 patch_prober.go:29] interesting pod/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 01 15:44:44.084119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:44.084018 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 01 15:44:44.096914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:44.096888 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2031217-46lng"] Nov 01 15:44:45.080473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:45.080439 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c085104-f3de-4d3f-abc3-3be9c57585d3" path="/var/lib/kubelet/pods/6c085104-f3de-4d3f-abc3-3be9c57585d3/volumes" Nov 01 15:44:45.083718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:45.083696 5473 patch_prober.go:29] interesting pod/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 01 15:44:45.083782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:45.083744 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 01 15:44:46.083491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:46.083456 5473 patch_prober.go:29] interesting pod/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 01 15:44:46.083898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:46.083510 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 01 15:44:47.082812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:47.082775 5473 patch_prober.go:29] interesting pod/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 01 15:44:47.083001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:47.082840 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 01 15:44:47.605972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:47.605928 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:47.943097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:47.942168 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:48.083135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:48.083099 5473 patch_prober.go:29] interesting pod/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" start-of-body= Nov 01 15:44:48.083338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:48.083155 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.101:15021/healthz/ready\": dial tcp 10.134.1.101:15021: connect: connection refused" Nov 01 15:44:49.088279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:49.088251 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:49.089413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:49.089391 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:44:52.200202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:52.200102 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:54.777652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:54.777620 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:54.933404 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:54.933371 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:55.048999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.048919 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f"] Nov 01 15:44:55.098838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.098805 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bde8715-cfm8f"] Nov 01 15:44:55.423088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.423006 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z"] Nov 01 15:44:55.426582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.426566 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.452661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.452628 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sshpw\" (UniqueName: \"kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.452854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.452730 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.452854 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.452768 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.480440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.480414 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:44:55.529856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.529824 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:44:55.532671 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.532647 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z"] Nov 01 15:44:55.553809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.553778 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.553942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.553818 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.553942 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.553869 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sshpw\" (UniqueName: \"kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.554563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.554541 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.556086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.556067 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.610917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.610891 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sshpw\" (UniqueName: \"kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw\") pod \"maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:55.739840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:55.739767 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:56.107479 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:44:56.107433 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e8bf5d_1bbf_4149_9c9c_feb94e87a893.slice/crio-4c8998b83ac604c22f848c0d9e43cbe44d20677281674b34c983021ff421432a WatchSource:0}: Error finding container 4c8998b83ac604c22f848c0d9e43cbe44d20677281674b34c983021ff421432a: Status 404 returned error can't find the container with id 4c8998b83ac604c22f848c0d9e43cbe44d20677281674b34c983021ff421432a Nov 01 15:44:56.110727 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:56.110704 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z"] Nov 01 15:44:56.229225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:56.229189 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" event={"ID":"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893","Type":"ContainerStarted","Data":"4c8998b83ac604c22f848c0d9e43cbe44d20677281674b34c983021ff421432a"} Nov 01 15:44:56.613508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:56.613478 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:44:57.079798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:57.079763 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e554956-20a3-4455-a3a0-06765c21a8ac" path="/var/lib/kubelet/pods/8e554956-20a3-4455-a3a0-06765c21a8ac/volumes" Nov 01 15:44:57.236376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:57.236336 5473 generic.go:356] "Generic (PLEG): container finished" podID="c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" containerID="1412a24b1097e427a3460c5cbb491dfd05faf4437d251ead5aba9d1e72ce1aa1" exitCode=0 Nov 01 15:44:57.236759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:57.236416 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" event={"ID":"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893","Type":"ContainerDied","Data":"1412a24b1097e427a3460c5cbb491dfd05faf4437d251ead5aba9d1e72ce1aa1"} Nov 01 15:44:58.048503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.048467 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv"] Nov 01 15:44:58.098226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.098189 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-245b55db-sjdgv"] Nov 01 15:44:58.379111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.379087 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:58.483818 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.483790 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw\" (UniqueName: \"kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw\") pod \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " Nov 01 15:44:58.483950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.483843 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca\") pod \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " Nov 01 15:44:58.483950 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.483915 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca\") pod \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\" (UID: \"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893\") " Nov 01 15:44:58.484253 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.484204 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca" pod "c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" (UID: "c5e8bf5d-1bbf-4149-9c9c-feb94e87a893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:44:58.485875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.485849 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca" pod "c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" (UID: "c5e8bf5d-1bbf-4149-9c9c-feb94e87a893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:44:58.485944 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.485891 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw" pod "c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" (UID: "c5e8bf5d-1bbf-4149-9c9c-feb94e87a893"). InnerVolumeSpecName "kube-api-access-sshpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:44:58.584848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.584821 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:58.584848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.584846 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sshpw\" (UniqueName: \"kubernetes.io/projected/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-kube-api-access-sshpw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:58.585028 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:58.584857 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:44:59.080301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:59.080267 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de84e4a-7a88-480a-9685-670fb0ac47a9" path="/var/lib/kubelet/pods/6de84e4a-7a88-480a-9685-670fb0ac47a9/volumes" Nov 01 15:44:59.250226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:59.250189 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" Nov 01 15:44:59.250414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:59.250189 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z" event={"ID":"c5e8bf5d-1bbf-4149-9c9c-feb94e87a893","Type":"ContainerDied","Data":"4c8998b83ac604c22f848c0d9e43cbe44d20677281674b34c983021ff421432a"} Nov 01 15:44:59.250414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:44:59.250340 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8998b83ac604c22f848c0d9e43cbe44d20677281674b34c983021ff421432a" Nov 01 15:45:00.156510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.156464 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9"] Nov 01 15:45:00.157019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.156897 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" containerName="revoke-keys" Nov 01 15:45:00.157019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.156909 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" containerName="revoke-keys" Nov 01 15:45:00.157019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.156980 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" containerName="revoke-keys" Nov 01 15:45:00.161577 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.161552 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.197660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.197530 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdrb\" (UniqueName: \"kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb\") pod \"maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.197660 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.197581 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.213862 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.213835 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:45:00.216349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.216329 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9"] Nov 01 15:45:00.298443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.298412 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.298620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.298533 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdrb\" (UniqueName: \"kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb\") pod \"maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.299037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.299017 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.355398 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.355370 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdrb\" (UniqueName: \"kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb\") pod \"maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.474386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.474302 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:00.839980 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:45:00.839949 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2010aa0_c47d_403c_b665_155d61454022.slice/crio-f7c8849af3f138f04b2440c99a0ef9df3d0b0727d8cccd7a955f88f395b6b28e WatchSource:0}: Error finding container f7c8849af3f138f04b2440c99a0ef9df3d0b0727d8cccd7a955f88f395b6b28e: Status 404 returned error can't find the container with id f7c8849af3f138f04b2440c99a0ef9df3d0b0727d8cccd7a955f88f395b6b28e Nov 01 15:45:00.842895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:00.842796 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9"] Nov 01 15:45:01.088459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:01.088388 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:01.108304 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:01.108266 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:45:01.265285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:01.265243 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" event={"ID":"b2010aa0-c47d-403c-b665-155d61454022","Type":"ContainerStarted","Data":"fa54f784bdb964e2468aed2d9cc5d1ee999ead4486831fa8b476b06af3389b62"} Nov 01 15:45:01.265702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:01.265311 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" event={"ID":"b2010aa0-c47d-403c-b665-155d61454022","Type":"ContainerStarted","Data":"f7c8849af3f138f04b2440c99a0ef9df3d0b0727d8cccd7a955f88f395b6b28e"} Nov 01 15:45:01.379300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:01.379182 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" podStartSLOduration=1.379169404 podStartE2EDuration="1.379169404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:45:01.37682102 +0000 UTC m=+5240.434972869" watchObservedRunningTime="2026-11-01 15:45:01.379169404 +0000 UTC m=+5240.437321292" Nov 01 15:45:01.550188 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:01.550153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:02.272460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:02.272427 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2010aa0-c47d-403c-b665-155d61454022" containerID="fa54f784bdb964e2468aed2d9cc5d1ee999ead4486831fa8b476b06af3389b62" exitCode=60 Nov 01 15:45:02.272879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:02.272514 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" event={"ID":"b2010aa0-c47d-403c-b665-155d61454022","Type":"ContainerDied","Data":"fa54f784bdb964e2468aed2d9cc5d1ee999ead4486831fa8b476b06af3389b62"} Nov 01 15:45:02.272879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:02.272679 5473 scope.go:122] "RemoveContainer" containerID="fa54f784bdb964e2468aed2d9cc5d1ee999ead4486831fa8b476b06af3389b62" Nov 01 15:45:03.280324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:03.280295 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2010aa0-c47d-403c-b665-155d61454022" containerID="c1aadf6c610f38e315833be33ebfd68bd0537f68aed2bce4506720634f3e0788" exitCode=60 Nov 01 15:45:03.280770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:03.280390 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" event={"ID":"b2010aa0-c47d-403c-b665-155d61454022","Type":"ContainerDied","Data":"c1aadf6c610f38e315833be33ebfd68bd0537f68aed2bce4506720634f3e0788"} Nov 01 15:45:03.280770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:03.280451 5473 scope.go:122] "RemoveContainer" containerID="fa54f784bdb964e2468aed2d9cc5d1ee999ead4486831fa8b476b06af3389b62" Nov 01 15:45:03.280770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:03.280624 5473 scope.go:122] "RemoveContainer" containerID="c1aadf6c610f38e315833be33ebfd68bd0537f68aed2bce4506720634f3e0788" Nov 01 15:45:03.280922 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:03.280894 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9_odh-ai-gateway-infra(b2010aa0-c47d-403c-b665-155d61454022)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" podUID="b2010aa0-c47d-403c-b665-155d61454022" Nov 01 15:45:04.288162 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:04.288129 5473 scope.go:122] "RemoveContainer" containerID="c1aadf6c610f38e315833be33ebfd68bd0537f68aed2bce4506720634f3e0788" Nov 01 15:45:04.288584 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:04.288400 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9_odh-ai-gateway-infra(b2010aa0-c47d-403c-b665-155d61454022)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" podUID="b2010aa0-c47d-403c-b665-155d61454022" Nov 01 15:45:05.501073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:05.501002 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:06.803924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:06.803892 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh"] Nov 01 15:45:06.909396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:06.909365 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb"] Nov 01 15:45:07.159844 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:07.159768 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:07.311786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:07.311756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:09.361934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.361902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:09.404869 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.404837 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9"] Nov 01 15:45:09.540530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.540501 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:09.679130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.679044 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca\") pod \"b2010aa0-c47d-403c-b665-155d61454022\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " Nov 01 15:45:09.679305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.679271 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb\" (UniqueName: \"kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb\") pod \"b2010aa0-c47d-403c-b665-155d61454022\" (UID: \"b2010aa0-c47d-403c-b665-155d61454022\") " Nov 01 15:45:09.679399 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.679378 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca" pod "b2010aa0-c47d-403c-b665-155d61454022" (UID: "b2010aa0-c47d-403c-b665-155d61454022"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:45:09.679565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.679551 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2010aa0-c47d-403c-b665-155d61454022-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:09.681450 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.681430 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb" pod "b2010aa0-c47d-403c-b665-155d61454022" (UID: "b2010aa0-c47d-403c-b665-155d61454022"). InnerVolumeSpecName "kube-api-access-kfdrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:45:09.780749 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:09.780718 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdrb\" (UniqueName: \"kubernetes.io/projected/b2010aa0-c47d-403c-b665-155d61454022-kube-api-access-kfdrb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:10.324304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.324261 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" event={"ID":"b2010aa0-c47d-403c-b665-155d61454022","Type":"ContainerDied","Data":"f7c8849af3f138f04b2440c99a0ef9df3d0b0727d8cccd7a955f88f395b6b28e"} Nov 01 15:45:10.324490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.324339 5473 scope.go:122] "RemoveContainer" containerID="c1aadf6c610f38e315833be33ebfd68bd0537f68aed2bce4506720634f3e0788" Nov 01 15:45:10.324490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.324273 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9" Nov 01 15:45:10.488061 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.488028 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9"] Nov 01 15:45:10.537653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.537624 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-1489d8ef-29892465-j7tb9"] Nov 01 15:45:10.753533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.753464 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:10.926864 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:10.926834 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:11.047450 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.047420 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz"] Nov 01 15:45:11.051368 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.051345 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4f15bc-np2tz"] Nov 01 15:45:11.080307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.080283 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08493723-38e8-4c45-8b92-ba2ec8d542cb" path="/var/lib/kubelet/pods/08493723-38e8-4c45-8b92-ba2ec8d542cb/volumes" Nov 01 15:45:11.080682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.080669 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2010aa0-c47d-403c-b665-155d61454022" path="/var/lib/kubelet/pods/b2010aa0-c47d-403c-b665-155d61454022/volumes" Nov 01 15:45:11.083199 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.083159 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:11.183966 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.183935 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z"] Nov 01 15:45:11.187515 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.187494 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1489d8ef-8pt5z"] Nov 01 15:45:11.804787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.804750 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" podUID="4e57fb0e-1743-4cdd-be95-2b5959d875ab" containerName="payload-processing" containerID="cri-o://7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff" gracePeriod=25 Nov 01 15:45:11.910579 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:11.910535 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" podUID="19772580-8033-4e11-9b86-70f37d555d47" containerName="payload-pre-processing" containerID="cri-o://e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd" gracePeriod=25 Nov 01 15:45:12.051437 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.051414 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:45:12.099638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.099612 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume\") pod \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " Nov 01 15:45:12.099771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.099703 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl\" (UniqueName: \"kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl\") pod \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\" (UID: \"4e57fb0e-1743-4cdd-be95-2b5959d875ab\") " Nov 01 15:45:12.100037 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.100005 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume" pod "4e57fb0e-1743-4cdd-be95-2b5959d875ab" (UID: "4e57fb0e-1743-4cdd-be95-2b5959d875ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:45:12.100088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.100078 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e57fb0e-1743-4cdd-be95-2b5959d875ab-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:12.101814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.101788 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl" pod "4e57fb0e-1743-4cdd-be95-2b5959d875ab" (UID: "4e57fb0e-1743-4cdd-be95-2b5959d875ab"). InnerVolumeSpecName "kube-api-access-d7xzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:45:12.156197 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.156173 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:45:12.200646 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.200619 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7xzl\" (UniqueName: \"kubernetes.io/projected/4e57fb0e-1743-4cdd-be95-2b5959d875ab-kube-api-access-d7xzl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:12.301586 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.301553 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume\") pod \"19772580-8033-4e11-9b86-70f37d555d47\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " Nov 01 15:45:12.301776 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.301613 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66\" (UniqueName: \"kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66\") pod \"19772580-8033-4e11-9b86-70f37d555d47\" (UID: \"19772580-8033-4e11-9b86-70f37d555d47\") " Nov 01 15:45:12.301912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.301889 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume" pod "19772580-8033-4e11-9b86-70f37d555d47" (UID: "19772580-8033-4e11-9b86-70f37d555d47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:45:12.301988 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.301973 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19772580-8033-4e11-9b86-70f37d555d47-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:12.303661 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.303640 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66" pod "19772580-8033-4e11-9b86-70f37d555d47" (UID: "19772580-8033-4e11-9b86-70f37d555d47"). InnerVolumeSpecName "kube-api-access-mkh66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:45:12.336877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.336816 5473 generic.go:356] "Generic (PLEG): container finished" podID="4e57fb0e-1743-4cdd-be95-2b5959d875ab" containerID="7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff" exitCode=0 Nov 01 15:45:12.336877 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.336863 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" Nov 01 15:45:12.337034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.336906 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" event={"ID":"4e57fb0e-1743-4cdd-be95-2b5959d875ab","Type":"ContainerDied","Data":"7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff"} Nov 01 15:45:12.337034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.336955 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh" event={"ID":"4e57fb0e-1743-4cdd-be95-2b5959d875ab","Type":"ContainerDied","Data":"1d2bbbd4f3d91d1f90d41dbb7335dddc231a26a864717ac6d94e4a30dd9f3fe6"} Nov 01 15:45:12.337034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.336980 5473 scope.go:122] "RemoveContainer" containerID="7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff" Nov 01 15:45:12.338339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.338316 5473 generic.go:356] "Generic (PLEG): container finished" podID="19772580-8033-4e11-9b86-70f37d555d47" containerID="e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd" exitCode=0 Nov 01 15:45:12.338456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.338344 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" event={"ID":"19772580-8033-4e11-9b86-70f37d555d47","Type":"ContainerDied","Data":"e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd"} Nov 01 15:45:12.338456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.338363 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" Nov 01 15:45:12.338456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.338395 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb" event={"ID":"19772580-8033-4e11-9b86-70f37d555d47","Type":"ContainerDied","Data":"b4637f2b13aeb8574eca5888e88a331c9542ea881a2a89d2a8dbe5f32e81dc32"} Nov 01 15:45:12.347172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.347150 5473 scope.go:122] "RemoveContainer" containerID="7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff" Nov 01 15:45:12.347468 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:12.347450 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff\": container with ID starting with 7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff not found: ID does not exist" containerID="7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff" Nov 01 15:45:12.347532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.347477 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff"} err="failed to get container status \"7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff\": rpc error: code = NotFound desc = could not find container \"7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff\": container with ID starting with 7cc2b59393ca525143b8131c27284390535b80fa076816345b11f5d4cc183eff not found: ID does not exist" Nov 01 15:45:12.347532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.347493 5473 scope.go:122] "RemoveContainer" containerID="e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd" Nov 01 15:45:12.358805 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.358781 5473 scope.go:122] "RemoveContainer" containerID="e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd" Nov 01 15:45:12.359088 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:12.359049 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd\": container with ID starting with e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd not found: ID does not exist" containerID="e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd" Nov 01 15:45:12.359173 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.359097 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd"} err="failed to get container status \"e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd\": rpc error: code = NotFound desc = could not find container \"e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd\": container with ID starting with e8025f90f9a982a2129c6293c54b9f43766c8c0ae0a098f3bf729023066dc0dd not found: ID does not exist" Nov 01 15:45:12.402786 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.402761 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkh66\" (UniqueName: \"kubernetes.io/projected/19772580-8033-4e11-9b86-70f37d555d47-kube-api-access-mkh66\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:12.484550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.484520 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:12.501171 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.501142 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb"] Nov 01 15:45:12.504926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.504905 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1489d8ef-64ffd7b455-gkwcb"] Nov 01 15:45:12.613866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.613801 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh"] Nov 01 15:45:12.615808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:12.615791 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1489d8ef-54b889d569-xvzmh"] Nov 01 15:45:13.080090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:13.080059 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19772580-8033-4e11-9b86-70f37d555d47" path="/var/lib/kubelet/pods/19772580-8033-4e11-9b86-70f37d555d47/volumes" Nov 01 15:45:13.080503 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:13.080474 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e57fb0e-1743-4cdd-be95-2b5959d875ab" path="/var/lib/kubelet/pods/4e57fb0e-1743-4cdd-be95-2b5959d875ab/volumes" Nov 01 15:45:13.080850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:13.080837 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e8bf5d-1bbf-4149-9c9c-feb94e87a893" path="/var/lib/kubelet/pods/c5e8bf5d-1bbf-4149-9c9c-feb94e87a893/volumes" Nov 01 15:45:16.539484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:16.539451 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr"] Nov 01 15:45:16.539873 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:16.539677 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" containerID="cri-o://51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586" gracePeriod=30 Nov 01 15:45:16.586848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:16.586818 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:18.046466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:18.046433 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78"] Nov 01 15:45:18.095917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:18.095890 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-550acb42-sjc78"] Nov 01 15:45:19.080228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:19.080187 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9030abb-d4c8-4da4-9460-9ae73221a9eb" path="/var/lib/kubelet/pods/f9030abb-d4c8-4da4-9460-9ae73221a9eb/volumes" Nov 01 15:45:19.091225 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:19.091187 5473 patch_prober.go:29] interesting pod/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:45:19.091359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:19.091264 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:45:21.783777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.783747 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:45:21.886708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886630 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.886708 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886678 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.886931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886717 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.886931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886764 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.886931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886816 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.886931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886896 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:45:21.886931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886900 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886949 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.886981 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887036 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68\") pod \"389eaa90-162b-477b-876c-09cf48f97de8\" (UID: \"389eaa90-162b-477b-876c-09cf48f97de8\") " Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887186 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887383 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887436 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887446 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887634 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887655 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389eaa90-162b-477b-876c-09cf48f97de8-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.887666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887673 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.888315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887686 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.888315 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.887699 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.889117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.889094 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:45:21.889373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.889342 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:45:21.889573 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.889549 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:45:21.889761 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.889738 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68" pod "389eaa90-162b-477b-876c-09cf48f97de8" (UID: "389eaa90-162b-477b-876c-09cf48f97de8"). InnerVolumeSpecName "kube-api-access-lph68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:45:21.989151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.989119 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.989151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.989150 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389eaa90-162b-477b-876c-09cf48f97de8-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.989344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.989163 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389eaa90-162b-477b-876c-09cf48f97de8-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:21.989344 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:21.989177 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lph68\" (UniqueName: \"kubernetes.io/projected/389eaa90-162b-477b-876c-09cf48f97de8-kube-api-access-lph68\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:22.404004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.403961 5473 generic.go:356] "Generic (PLEG): container finished" podID="389eaa90-162b-477b-876c-09cf48f97de8" containerID="51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586" exitCode=0 Nov 01 15:45:22.404226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.404025 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" Nov 01 15:45:22.404226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.404046 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" event={"ID":"389eaa90-162b-477b-876c-09cf48f97de8","Type":"ContainerDied","Data":"51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586"} Nov 01 15:45:22.404226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.404114 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr" event={"ID":"389eaa90-162b-477b-876c-09cf48f97de8","Type":"ContainerDied","Data":"4cadcec85907baa44b57ed65f38b4bd5dad9c71937dd438a6d889bc427b98546"} Nov 01 15:45:22.404226 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.404144 5473 scope.go:122] "RemoveContainer" containerID="51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586" Nov 01 15:45:22.414150 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.414134 5473 scope.go:122] "RemoveContainer" containerID="b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36" Nov 01 15:45:22.422702 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.422681 5473 scope.go:122] "RemoveContainer" containerID="51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586" Nov 01 15:45:22.422934 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:22.422915 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586\": container with ID starting with 51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586 not found: ID does not exist" containerID="51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586" Nov 01 15:45:22.423002 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.422941 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586"} err="failed to get container status \"51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586\": rpc error: code = NotFound desc = could not find container \"51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586\": container with ID starting with 51b92045f72ab7b86f172089f63269169f8214daba6fa80789266d6eaebeb586 not found: ID does not exist" Nov 01 15:45:22.423002 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.422963 5473 scope.go:122] "RemoveContainer" containerID="b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36" Nov 01 15:45:22.423249 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:22.423232 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36\": container with ID starting with b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36 not found: ID does not exist" containerID="b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36" Nov 01 15:45:22.423312 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.423253 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36"} err="failed to get container status \"b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36\": rpc error: code = NotFound desc = could not find container \"b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36\": container with ID starting with b1d25f0aae82e785d45b2481df1695e401fb66fb0de8ebe68e6f8de00104fa36 not found: ID does not exist" Nov 01 15:45:22.569803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.569772 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr"] Nov 01 15:45:22.619803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:22.619770 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1489d8ef-openshift-default-6cccfc5f99-s2fdr"] Nov 01 15:45:23.081019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:23.080985 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389eaa90-162b-477b-876c-09cf48f97de8" path="/var/lib/kubelet/pods/389eaa90-162b-477b-876c-09cf48f97de8/volumes" Nov 01 15:45:29.047733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:29.047696 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj"] Nov 01 15:45:29.079770 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:29.079742 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f509b530-v24xj"] Nov 01 15:45:31.080157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:31.080122 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486e25f6-eabc-4114-8d18-5ab19e6868f8" path="/var/lib/kubelet/pods/486e25f6-eabc-4114-8d18-5ab19e6868f8/volumes" Nov 01 15:45:31.106721 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:45:31.106689 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:45:43.852650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.852616 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72"] Nov 01 15:45:43.853052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.852999 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2010aa0-c47d-403c-b665-155d61454022" containerName="cleanup" Nov 01 15:45:43.853052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853011 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2010aa0-c47d-403c-b665-155d61454022" containerName="cleanup" Nov 01 15:45:43.853052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853024 5473 cpu_manager.go:414] "containerMap: removing container" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" Nov 01 15:45:43.853052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853044 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853056 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2010aa0-c47d-403c-b665-155d61454022" containerName="cleanup" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853061 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2010aa0-c47d-403c-b665-155d61454022" containerName="cleanup" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853069 5473 cpu_manager.go:414] "containerMap: removing container" podUID="19772580-8033-4e11-9b86-70f37d555d47" containerName="payload-pre-processing" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853074 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19772580-8033-4e11-9b86-70f37d555d47" containerName="payload-pre-processing" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853082 5473 cpu_manager.go:414] "containerMap: removing container" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853087 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853097 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4e57fb0e-1743-4cdd-be95-2b5959d875ab" containerName="payload-processing" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853102 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e57fb0e-1743-4cdd-be95-2b5959d875ab" containerName="payload-processing" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853155 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e57fb0e-1743-4cdd-be95-2b5959d875ab" containerName="payload-processing" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853167 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2010aa0-c47d-403c-b665-155d61454022" containerName="cleanup" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853173 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853179 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2010aa0-c47d-403c-b665-155d61454022" containerName="cleanup" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853184 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="19772580-8033-4e11-9b86-70f37d555d47" containerName="payload-pre-processing" Nov 01 15:45:43.853203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.853190 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="389eaa90-162b-477b-876c-09cf48f97de8" containerName="istio-proxy" Nov 01 15:45:43.856620 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.856594 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.882153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.882124 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.882292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.882160 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.882292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.882185 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs7j\" (UniqueName: \"kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.913390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.913361 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72"] Nov 01 15:45:43.960323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.960296 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:45:43.960323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.960318 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:45:43.983599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.983575 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.983742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.983609 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.983742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.983639 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs7j\" (UniqueName: \"kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.984259 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.984237 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:43.985981 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:43.985961 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:44.039921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:44.039896 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs7j\" (UniqueName: \"kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j\") pod \"maas-api-revoke-sub-85237e45-mtm72\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:44.170171 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:44.170115 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:44.541772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:44.541740 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72"] Nov 01 15:45:44.543281 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:45:44.543251 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062214c0_62a5_4fcb_bd46_22e0a21ddbb7.slice/crio-19562e5e522d8c952c4362f6507edd751927d80fbf131037760756092a3f0a3e WatchSource:0}: Error finding container 19562e5e522d8c952c4362f6507edd751927d80fbf131037760756092a3f0a3e: Status 404 returned error can't find the container with id 19562e5e522d8c952c4362f6507edd751927d80fbf131037760756092a3f0a3e Nov 01 15:45:45.047400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.047368 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678"] Nov 01 15:45:45.050383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.050355 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44aeabe8-ld678"] Nov 01 15:45:45.068561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.068537 5473 scope.go:122] "RemoveContainer" containerID="10f3c48ced753d95412ca5c6581eef9d980a909c470219d9358b489a6c6922b7" Nov 01 15:45:45.078238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.078208 5473 scope.go:122] "RemoveContainer" containerID="cb86d055379ded9b28a7564ae9495fb81523cedc47ff9575f0b6db254845988f" Nov 01 15:45:45.080687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.080661 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ab152a-f149-4057-be2d-f6ba2d9a9b45" path="/var/lib/kubelet/pods/c8ab152a-f149-4057-be2d-f6ba2d9a9b45/volumes" Nov 01 15:45:45.087096 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.087079 5473 scope.go:122] "RemoveContainer" containerID="522c143ec4ba03f47772ec6e31a029c1524baa7d7e4e8681bb026f247a6290d8" Nov 01 15:45:45.095679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.095660 5473 scope.go:122] "RemoveContainer" containerID="267444ac40320d77196baa6a39cec43d7b663bdd45cc5d07ade5df85a2f1a003" Nov 01 15:45:45.103651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.103625 5473 scope.go:122] "RemoveContainer" containerID="5b1874c2d6591a76441d9f6aa3eeffeb63ba921efd0a43cb1c67e14b5fe2d816" Nov 01 15:45:45.111555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.111530 5473 scope.go:122] "RemoveContainer" containerID="3014fcd59ff7f07942cc3621299731e57936522109ba93937092118fe3960f22" Nov 01 15:45:45.119400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.119376 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9"] Nov 01 15:45:45.124991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.124967 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.127980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.127955 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg"] Nov 01 15:45:45.133008 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.132814 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.178768 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.178742 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 15:45:45.179000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.178982 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-58vzm\"" Nov 01 15:45:45.181757 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.181734 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9"] Nov 01 15:45:45.196135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196108 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196150 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196179 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196335 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196300 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a84ef62-7215-4e81-8585-2ed334e28a34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196399 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552f992f-89c3-43bf-8a2c-44665d612997-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196434 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twnj\" (UniqueName: \"kubernetes.io/projected/552f992f-89c3-43bf-8a2c-44665d612997-kube-api-access-8twnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196468 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196505 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196541 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196603 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196641 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196675 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552f992f-89c3-43bf-8a2c-44665d612997-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196701 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.196878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196766 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196785 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196814 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196836 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bw5c\" (UniqueName: \"kubernetes.io/projected/8a84ef62-7215-4e81-8585-2ed334e28a34-kube-api-access-9bw5c\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.196878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.196857 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552f992f-89c3-43bf-8a2c-44665d612997-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.230608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.230579 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg"] Nov 01 15:45:45.297719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297687 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552f992f-89c3-43bf-8a2c-44665d612997-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.297719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297719 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.297968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297754 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.297968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297875 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.297968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297925 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.297968 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297962 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bw5c\" (UniqueName: \"kubernetes.io/projected/8a84ef62-7215-4e81-8585-2ed334e28a34-kube-api-access-9bw5c\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.298206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.297998 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552f992f-89c3-43bf-8a2c-44665d612997-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298038 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298068 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298093 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298149 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a84ef62-7215-4e81-8585-2ed334e28a34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.298206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298196 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552f992f-89c3-43bf-8a2c-44665d612997-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298250 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twnj\" (UniqueName: \"kubernetes.io/projected/552f992f-89c3-43bf-8a2c-44665d612997-kube-api-access-8twnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298290 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.298601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298318 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.298601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298348 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.298601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298383 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.298601 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298423 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298606 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298864 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298961 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.298991 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.299257 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.299307 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.299523 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/552f992f-89c3-43bf-8a2c-44665d612997-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.299530 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.299892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.299726 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.300374 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.299929 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a84ef62-7215-4e81-8585-2ed334e28a34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.300442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.300389 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/552f992f-89c3-43bf-8a2c-44665d612997-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.300484 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.300443 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.300711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.300692 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/552f992f-89c3-43bf-8a2c-44665d612997-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.300772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.300756 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.354690 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.354632 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bw5c\" (UniqueName: \"kubernetes.io/projected/8a84ef62-7215-4e81-8585-2ed334e28a34-kube-api-access-9bw5c\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.358475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.358450 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a84ef62-7215-4e81-8585-2ed334e28a34-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-56jt9\" (UID: \"8a84ef62-7215-4e81-8585-2ed334e28a34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.359308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.359286 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twnj\" (UniqueName: \"kubernetes.io/projected/552f992f-89c3-43bf-8a2c-44665d612997-kube-api-access-8twnj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.359733 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.359711 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/552f992f-89c3-43bf-8a2c-44665d612997-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-p8scg\" (UID: \"552f992f-89c3-43bf-8a2c-44665d612997\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.442066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.442034 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:45.451311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.451260 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:45.538710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.538679 5473 generic.go:356] "Generic (PLEG): container finished" podID="062214c0-62a5-4fcb-bd46-22e0a21ddbb7" containerID="80aff74cd0c2d5b04d147bacb8219f962d3f6ce5fdd85d6992f243ce6221ba2b" exitCode=0 Nov 01 15:45:45.538837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.538723 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" event={"ID":"062214c0-62a5-4fcb-bd46-22e0a21ddbb7","Type":"ContainerDied","Data":"80aff74cd0c2d5b04d147bacb8219f962d3f6ce5fdd85d6992f243ce6221ba2b"} Nov 01 15:45:45.538837 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.538774 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" event={"ID":"062214c0-62a5-4fcb-bd46-22e0a21ddbb7","Type":"ContainerStarted","Data":"19562e5e522d8c952c4362f6507edd751927d80fbf131037760756092a3f0a3e"} Nov 01 15:45:45.872189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.872157 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9"] Nov 01 15:45:45.873127 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:45:45.873097 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a84ef62_7215_4e81_8585_2ed334e28a34.slice/crio-a89bfd4eb549baceb6480780520bcdf10033318fc5d12402197413cec24b46cb WatchSource:0}: Error finding container a89bfd4eb549baceb6480780520bcdf10033318fc5d12402197413cec24b46cb: Status 404 returned error can't find the container with id a89bfd4eb549baceb6480780520bcdf10033318fc5d12402197413cec24b46cb Nov 01 15:45:45.874571 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.874546 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg"] Nov 01 15:45:45.875802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.875767 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:45:45.875866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.875848 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:45:45.875911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.875896 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:45:45.877954 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:45:45.877923 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552f992f_89c3_43bf_8a2c_44665d612997.slice/crio-c0cbe9e581c50eb9a62eaa203054c19fd89136eba24bdc9f9c130dbf52d369b2 WatchSource:0}: Error finding container c0cbe9e581c50eb9a62eaa203054c19fd89136eba24bdc9f9c130dbf52d369b2: Status 404 returned error can't find the container with id c0cbe9e581c50eb9a62eaa203054c19fd89136eba24bdc9f9c130dbf52d369b2 Nov 01 15:45:45.883796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.883768 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:45:45.883868 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.883841 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:45:45.883912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:45.883881 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:45:46.577243 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.577179 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" event={"ID":"8a84ef62-7215-4e81-8585-2ed334e28a34","Type":"ContainerStarted","Data":"2ced14ffdb2a8c7ff94f568c149f50c852b65a085cd7bd1b570a55955f9a9620"} Nov 01 15:45:46.577799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.577270 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" event={"ID":"8a84ef62-7215-4e81-8585-2ed334e28a34","Type":"ContainerStarted","Data":"a89bfd4eb549baceb6480780520bcdf10033318fc5d12402197413cec24b46cb"} Nov 01 15:45:46.580269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.580232 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" event={"ID":"552f992f-89c3-43bf-8a2c-44665d612997","Type":"ContainerStarted","Data":"8f10738558d90b364b6ccd0cf7fe545dfc06649291dbc63eb0d3f3752cdec76f"} Nov 01 15:45:46.580430 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.580285 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" event={"ID":"552f992f-89c3-43bf-8a2c-44665d612997","Type":"ContainerStarted","Data":"c0cbe9e581c50eb9a62eaa203054c19fd89136eba24bdc9f9c130dbf52d369b2"} Nov 01 15:45:46.706406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.706343 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podStartSLOduration=1.706325318 podStartE2EDuration="1.706325318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:45:46.702403962 +0000 UTC m=+5285.760555812" watchObservedRunningTime="2026-11-01 15:45:46.706325318 +0000 UTC m=+5285.764477168" Nov 01 15:45:46.800798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.800771 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:46.816976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.816947 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca\") pod \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " Nov 01 15:45:46.817120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.817037 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j\" (UniqueName: \"kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j\") pod \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " Nov 01 15:45:46.817164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.817150 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca\") pod \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\" (UID: \"062214c0-62a5-4fcb-bd46-22e0a21ddbb7\") " Nov 01 15:45:46.817740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.817466 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca" pod "062214c0-62a5-4fcb-bd46-22e0a21ddbb7" (UID: "062214c0-62a5-4fcb-bd46-22e0a21ddbb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:45:46.817740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.817595 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:46.822185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.821826 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podStartSLOduration=1.821803753 podStartE2EDuration="1.821803753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:45:46.821402149 +0000 UTC m=+5285.879553999" watchObservedRunningTime="2026-11-01 15:45:46.821803753 +0000 UTC m=+5285.879955597" Nov 01 15:45:46.822185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.822009 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca" pod "062214c0-62a5-4fcb-bd46-22e0a21ddbb7" (UID: "062214c0-62a5-4fcb-bd46-22e0a21ddbb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:45:46.822357 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.822260 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j" pod "062214c0-62a5-4fcb-bd46-22e0a21ddbb7" (UID: "062214c0-62a5-4fcb-bd46-22e0a21ddbb7"). InnerVolumeSpecName "kube-api-access-kbs7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:45:46.918422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.918385 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:46.918422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:46.918417 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbs7j\" (UniqueName: \"kubernetes.io/projected/062214c0-62a5-4fcb-bd46-22e0a21ddbb7-kube-api-access-kbs7j\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:45:47.446238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.445635 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:47.450240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.447856 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 01 15:45:47.450240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.447919 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 01 15:45:47.454539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.454517 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:47.458236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.456460 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" start-of-body= Nov 01 15:45:47.458236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.456514 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" Nov 01 15:45:47.589582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.589548 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" Nov 01 15:45:47.589978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.589947 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72" event={"ID":"062214c0-62a5-4fcb-bd46-22e0a21ddbb7","Type":"ContainerDied","Data":"19562e5e522d8c952c4362f6507edd751927d80fbf131037760756092a3f0a3e"} Nov 01 15:45:47.590294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:47.590275 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19562e5e522d8c952c4362f6507edd751927d80fbf131037760756092a3f0a3e" Nov 01 15:45:48.446240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:48.445696 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 01 15:45:48.446240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:48.445766 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 01 15:45:48.461240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:48.454306 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" start-of-body= Nov 01 15:45:48.461240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:48.454364 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" Nov 01 15:45:49.449240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:49.442644 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 01 15:45:49.449240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:49.442712 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 01 15:45:49.456246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:49.455590 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" start-of-body= Nov 01 15:45:49.456246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:49.455644 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" Nov 01 15:45:50.443719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:50.443680 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 01 15:45:50.444014 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:50.443985 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 01 15:45:50.473345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:50.473305 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" start-of-body= Nov 01 15:45:50.473933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:50.473899 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" Nov 01 15:45:51.451241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:51.450556 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 01 15:45:51.451241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:51.450620 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 01 15:45:51.461876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:51.453622 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" start-of-body= Nov 01 15:45:51.461876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:51.453673 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" Nov 01 15:45:52.443149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:52.443111 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 01 15:45:52.443606 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:52.443181 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 01 15:45:52.452102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:52.452074 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" start-of-body= Nov 01 15:45:52.452281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:52.452127 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": dial tcp 10.134.1.108:15021: connect: connection refused" Nov 01 15:45:54.442843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:54.442794 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:45:54.443362 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:54.442871 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:45:54.453604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:54.453557 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:45:54.453764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:54.453615 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:45:55.484927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:55.484401 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:45:55.484927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:55.484494 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:45:55.495371 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:55.495343 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:45:55.495528 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:55.495398 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:45:56.206033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.205992 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:45:56.496929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.496828 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:45:56.496929 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.496909 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:45:56.518196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.518158 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:45:56.518358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.518210 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:45:56.531922 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.531899 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:56.532256 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.532211 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:56.532894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.532876 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" Nov 01 15:45:56.553053 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.553032 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:56.553265 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.553246 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:56.553927 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:56.553910 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" Nov 01 15:45:58.308711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:45:58.308675 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:46:01.106010 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:46:01.105974 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:46:11.281434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:11.281400 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:46:12.047845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:12.047807 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq"] Nov 01 15:46:12.098205 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:12.098175 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c61b32d-pjtrq"] Nov 01 15:46:13.080279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:13.080246 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29b786c-35d9-4eb5-be98-1dd51f5dac65" path="/var/lib/kubelet/pods/d29b786c-35d9-4eb5-be98-1dd51f5dac65/volumes" Nov 01 15:46:27.594417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:27.594372 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:46:27.594851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:27.594445 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:46:29.048773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:29.048736 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88"] Nov 01 15:46:29.098424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:29.098390 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab2218b8-c5f88"] Nov 01 15:46:31.081058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.081020 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f633d5b-4ebc-4718-b1ee-2e0f33baba07" path="/var/lib/kubelet/pods/0f633d5b-4ebc-4718-b1ee-2e0f33baba07/volumes" Nov 01 15:46:31.107156 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:46:31.107124 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:46:31.259460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.259425 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m"] Nov 01 15:46:31.259822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.259809 5473 cpu_manager.go:414] "containerMap: removing container" podUID="062214c0-62a5-4fcb-bd46-22e0a21ddbb7" containerName="revoke-keys" Nov 01 15:46:31.259870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.259823 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062214c0-62a5-4fcb-bd46-22e0a21ddbb7" containerName="revoke-keys" Nov 01 15:46:31.259907 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.259893 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="062214c0-62a5-4fcb-bd46-22e0a21ddbb7" containerName="revoke-keys" Nov 01 15:46:31.263105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.263089 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.293426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.293402 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkq6\" (UniqueName: \"kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.293541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.293493 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.293541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.293527 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.317341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.317315 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:46:31.367208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.367113 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:46:31.369459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.369433 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m"] Nov 01 15:46:31.394816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.394782 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkq6\" (UniqueName: \"kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.394998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.394843 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.394998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.394867 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.395545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.395519 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.397267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.397246 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.451959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.451930 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkq6\" (UniqueName: \"kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6\") pod \"maas-api-revoke-sub-f904bff9-lgm4m\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.576373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.576338 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:31.946978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:31.946949 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m"] Nov 01 15:46:31.948815 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:46:31.948785 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f6605c_9daf_4027_871d_2b216a0c42fb.slice/crio-26cb8a2738ca9f1f34c940cb40c75f50c7b04904716eb512fbff24c4b95f3e45 WatchSource:0}: Error finding container 26cb8a2738ca9f1f34c940cb40c75f50c7b04904716eb512fbff24c4b95f3e45: Status 404 returned error can't find the container with id 26cb8a2738ca9f1f34c940cb40c75f50c7b04904716eb512fbff24c4b95f3e45 Nov 01 15:46:32.847038 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:32.847001 5473 generic.go:356] "Generic (PLEG): container finished" podID="95f6605c-9daf-4027-871d-2b216a0c42fb" containerID="469b3d12fe1a8d2ac7ea8747954bbb42995d4e1b48f0f06013181e71bc96cc41" exitCode=0 Nov 01 15:46:32.847472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:32.847083 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" event={"ID":"95f6605c-9daf-4027-871d-2b216a0c42fb","Type":"ContainerDied","Data":"469b3d12fe1a8d2ac7ea8747954bbb42995d4e1b48f0f06013181e71bc96cc41"} Nov 01 15:46:32.847472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:32.847135 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" event={"ID":"95f6605c-9daf-4027-871d-2b216a0c42fb","Type":"ContainerStarted","Data":"26cb8a2738ca9f1f34c940cb40c75f50c7b04904716eb512fbff24c4b95f3e45"} Nov 01 15:46:33.978947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:33.978923 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:34.015092 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.015063 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca\") pod \"95f6605c-9daf-4027-871d-2b216a0c42fb\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " Nov 01 15:46:34.015241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.015133 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6\" (UniqueName: \"kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6\") pod \"95f6605c-9daf-4027-871d-2b216a0c42fb\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " Nov 01 15:46:34.015241 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.015180 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca\") pod \"95f6605c-9daf-4027-871d-2b216a0c42fb\" (UID: \"95f6605c-9daf-4027-871d-2b216a0c42fb\") " Nov 01 15:46:34.015486 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.015460 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca" pod "95f6605c-9daf-4027-871d-2b216a0c42fb" (UID: "95f6605c-9daf-4027-871d-2b216a0c42fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:46:34.017288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.017255 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6" pod "95f6605c-9daf-4027-871d-2b216a0c42fb" (UID: "95f6605c-9daf-4027-871d-2b216a0c42fb"). InnerVolumeSpecName "kube-api-access-9jkq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:46:34.017384 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.017297 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca" pod "95f6605c-9daf-4027-871d-2b216a0c42fb" (UID: "95f6605c-9daf-4027-871d-2b216a0c42fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:46:34.116365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.116298 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:46:34.116365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.116324 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95f6605c-9daf-4027-871d-2b216a0c42fb-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:46:34.116365 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.116334 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkq6\" (UniqueName: \"kubernetes.io/projected/95f6605c-9daf-4027-871d-2b216a0c42fb-kube-api-access-9jkq6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:46:34.860627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.860593 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" event={"ID":"95f6605c-9daf-4027-871d-2b216a0c42fb","Type":"ContainerDied","Data":"26cb8a2738ca9f1f34c940cb40c75f50c7b04904716eb512fbff24c4b95f3e45"} Nov 01 15:46:34.860627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.860627 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m" Nov 01 15:46:34.860841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:34.860671 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26cb8a2738ca9f1f34c940cb40c75f50c7b04904716eb512fbff24c4b95f3e45" Nov 01 15:46:45.115875 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.115844 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl"] Nov 01 15:46:45.116314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.116259 5473 cpu_manager.go:414] "containerMap: removing container" podUID="95f6605c-9daf-4027-871d-2b216a0c42fb" containerName="revoke-keys" Nov 01 15:46:45.116314 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.116273 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f6605c-9daf-4027-871d-2b216a0c42fb" containerName="revoke-keys" Nov 01 15:46:45.116401 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.116343 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f6605c-9daf-4027-871d-2b216a0c42fb" containerName="revoke-keys" Nov 01 15:46:45.119652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.119633 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.166487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.166463 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v"] Nov 01 15:46:45.169819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.169802 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.174301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.174279 5473 scope.go:122] "RemoveContainer" containerID="703d417231ef142492283b2c957f392895f43a92f1528e0496fce5f58bf71134" Nov 01 15:46:45.184146 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.184110 5473 scope.go:122] "RemoveContainer" containerID="99758dde9be5655cce225276845226870becefb1f30932f4dd6181513f0a0f12" Nov 01 15:46:45.192402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.192384 5473 scope.go:122] "RemoveContainer" containerID="1a31d5ed39a71c82e51f857037304718548b5d0e3c9b58c91cd9d9da7ff1c26e" Nov 01 15:46:45.200329 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.200311 5473 scope.go:122] "RemoveContainer" containerID="fbb129ab9b9ca0695dd7a528eadbd199ab3996990b24b0ab65c28b3e77478682" Nov 01 15:46:45.208710 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.208689 5473 scope.go:122] "RemoveContainer" containerID="7def863bfccf97e74d9e4538e534e4da9a0de6994304e55ed26570aecb97f082" Nov 01 15:46:45.216366 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.216337 5473 scope.go:122] "RemoveContainer" containerID="741ad7b253d7151ebea909b7ffdba3e353a03a790a3fb86adecc72d901380054" Nov 01 15:46:45.221531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221504 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.221635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221546 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.221683 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221655 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea2e96a-5411-4745-903f-275b69c10843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.221722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221711 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea2e96a-5411-4745-903f-275b69c10843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.221827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221781 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea2e96a-5411-4745-903f-275b69c10843-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.221882 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221853 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.221932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221880 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64gwt\" (UniqueName: \"kubernetes.io/projected/bdad6cc9-742d-42e6-a82d-b8c8664ed362-kube-api-access-64gwt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.221932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221914 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.222036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221960 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl"] Nov 01 15:46:45.222036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.221998 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.222112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222045 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.222112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222065 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t924j\" (UniqueName: \"kubernetes.io/projected/aea2e96a-5411-4745-903f-275b69c10843-kube-api-access-t924j\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.222112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222090 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.222112 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222106 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.222352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222125 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.222352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222165 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.222459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222419 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.222521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222478 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.222521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.222504 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.273203 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.273175 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v"] Nov 01 15:46:45.323233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323185 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea2e96a-5411-4745-903f-275b69c10843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323278 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea2e96a-5411-4745-903f-275b69c10843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323320 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea2e96a-5411-4745-903f-275b69c10843-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323349 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323375 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64gwt\" (UniqueName: \"kubernetes.io/projected/bdad6cc9-742d-42e6-a82d-b8c8664ed362-kube-api-access-64gwt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.323567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323408 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323436 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.323567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323461 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.323796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323621 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t924j\" (UniqueName: \"kubernetes.io/projected/aea2e96a-5411-4745-903f-275b69c10843-kube-api-access-t924j\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323690 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.323796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323721 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.323796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323772 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323800 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323843 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323899 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.324027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323926 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.324027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.323928 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.324003 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.324035 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.324361 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.324576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.324506 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.325191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.324755 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aea2e96a-5411-4745-903f-275b69c10843-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.325191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.324771 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.325428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.325260 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.325548 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.325524 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.325623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.325606 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.325684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.325609 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.326180 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.326150 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.327598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.327361 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aea2e96a-5411-4745-903f-275b69c10843-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.327773 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.327730 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.327884 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.327809 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aea2e96a-5411-4745-903f-275b69c10843-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.328477 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.328451 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.380104 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.380032 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aea2e96a-5411-4745-903f-275b69c10843-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.380379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.380359 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t924j\" (UniqueName: \"kubernetes.io/projected/aea2e96a-5411-4745-903f-275b69c10843-kube-api-access-t924j\") pod \"maas-default-gateway-openshift-default-7f5667f86c-l5vkl\" (UID: \"aea2e96a-5411-4745-903f-275b69c10843\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.380460 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.380408 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64gwt\" (UniqueName: \"kubernetes.io/projected/bdad6cc9-742d-42e6-a82d-b8c8664ed362-kube-api-access-64gwt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.399031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.399004 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdad6cc9-742d-42e6-a82d-b8c8664ed362-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-x769v\" (UID: \"bdad6cc9-742d-42e6-a82d-b8c8664ed362\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.433018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.432995 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:45.483767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.483734 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:45.856746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.856715 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl"] Nov 01 15:46:45.857909 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:46:45.857882 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea2e96a_5411_4745_903f_275b69c10843.slice/crio-e4bef0c83b9003e537ae6d57757070b45aa6881eada4e3a2a51b0622b656537f WatchSource:0}: Error finding container e4bef0c83b9003e537ae6d57757070b45aa6881eada4e3a2a51b0622b656537f: Status 404 returned error can't find the container with id e4bef0c83b9003e537ae6d57757070b45aa6881eada4e3a2a51b0622b656537f Nov 01 15:46:45.860540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.860505 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:46:45.860619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.860586 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:46:45.860619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.860615 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:46:45.907339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.907314 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v"] Nov 01 15:46:45.909927 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:46:45.909904 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdad6cc9_742d_42e6_a82d_b8c8664ed362.slice/crio-de628024a164b02063c241c4edad45d098d7b60c95944f6c9fa6d7a0b10887bd WatchSource:0}: Error finding container de628024a164b02063c241c4edad45d098d7b60c95944f6c9fa6d7a0b10887bd: Status 404 returned error can't find the container with id de628024a164b02063c241c4edad45d098d7b60c95944f6c9fa6d7a0b10887bd Nov 01 15:46:45.912386 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.912348 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:46:45.912470 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.912436 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:46:45.912512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.912482 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:46:45.933403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.933368 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" event={"ID":"aea2e96a-5411-4745-903f-275b69c10843","Type":"ContainerStarted","Data":"6b93f65a3e992a9a6909b0e1f716a37315123ebdbedfbff618284be32191ee57"} Nov 01 15:46:45.933524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.933423 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" event={"ID":"aea2e96a-5411-4745-903f-275b69c10843","Type":"ContainerStarted","Data":"e4bef0c83b9003e537ae6d57757070b45aa6881eada4e3a2a51b0622b656537f"} Nov 01 15:46:45.934827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:45.934800 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" event={"ID":"bdad6cc9-742d-42e6-a82d-b8c8664ed362","Type":"ContainerStarted","Data":"de628024a164b02063c241c4edad45d098d7b60c95944f6c9fa6d7a0b10887bd"} Nov 01 15:46:46.048917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:46.048865 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podStartSLOduration=1.048851459 podStartE2EDuration="1.048851459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:46.04870152 +0000 UTC m=+5345.106853394" watchObservedRunningTime="2026-11-01 15:46:46.048851459 +0000 UTC m=+5345.107003308" Nov 01 15:46:46.441240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:46.436589 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:46.456239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:46.455358 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:46.456239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:46.455440 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:46.973576 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:46.973532 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" event={"ID":"bdad6cc9-742d-42e6-a82d-b8c8664ed362","Type":"ContainerStarted","Data":"f2616106b65461a1bf7b4620b4b625fa4da0c964c81d17fe6f7c8de31f07fbbe"} Nov 01 15:46:47.190663 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:47.190616 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podStartSLOduration=2.190590176 podStartE2EDuration="2.190590176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:46:47.140988553 +0000 UTC m=+5346.199140405" watchObservedRunningTime="2026-11-01 15:46:47.190590176 +0000 UTC m=+5346.248742025" Nov 01 15:46:47.433709 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:47.433673 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:47.434007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:47.433959 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:47.484694 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:47.484663 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:47.486132 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:47.486100 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:47.486281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:47.486152 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:48.437177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:48.436671 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:48.437177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:48.436775 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:48.484772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:48.484732 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:48.485181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:48.484786 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:49.439239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:49.438657 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:49.439239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:49.438713 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:49.485240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:49.484619 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:49.485240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:49.484684 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:50.435240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:50.434726 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:50.435240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:50.434804 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:50.484650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:50.484610 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:50.484935 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:50.484905 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:51.436239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:51.435593 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:51.436239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:51.435655 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:51.486239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:51.485613 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:51.486239 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:51.485675 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:52.433732 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:52.433693 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:52.433918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:52.433763 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:52.485311 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:52.485274 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:52.485691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:52.485332 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:53.434288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:53.434254 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 01 15:46:53.434471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:53.434321 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 01 15:46:53.485143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:53.485107 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 01 15:46:53.485302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:53.485164 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 01 15:46:55.475422 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:55.475360 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:46:55.475974 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:55.475447 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:46:55.526415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:55.526364 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:46:55.526607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:55.526427 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:46:56.476343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.476246 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:46:56.476343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.476308 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:46:56.568408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.568356 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:46:56.568612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.568420 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:46:56.600080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.600048 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:56.600512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.600489 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:56.601236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.601200 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" Nov 01 15:46:56.759801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.759721 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:56.760021 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.760002 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:46:56.760617 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:46:56.760600 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" Nov 01 15:47:01.106876 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:47:01.106838 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:47:03.669527 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.669493 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn"] Nov 01 15:47:03.672812 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.672789 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.690743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.690716 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn"] Nov 01 15:47:03.715719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.715689 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd9v\" (UniqueName: \"kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.715874 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.715746 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.715954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.715903 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.731304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.731278 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:47:03.731478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.731336 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:47:03.817271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.817239 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd9v\" (UniqueName: \"kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.817447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.817284 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.817447 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.817354 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.818002 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.817978 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.819583 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.819563 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.873814 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.873791 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd9v\" (UniqueName: \"kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v\") pod \"maas-api-revoke-sub-f03629db-qpltn\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:03.988685 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:03.988623 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:04.358022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:04.357989 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn"] Nov 01 15:47:04.359540 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:47:04.359510 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7b2358_e2ca_4468_a192_c66914e53670.slice/crio-8bc00ed68446db5a25d2f5f551179f9e7a976ffbc0c5652ee60c651e38f05416 WatchSource:0}: Error finding container 8bc00ed68446db5a25d2f5f551179f9e7a976ffbc0c5652ee60c651e38f05416: Status 404 returned error can't find the container with id 8bc00ed68446db5a25d2f5f551179f9e7a976ffbc0c5652ee60c651e38f05416 Nov 01 15:47:05.141190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:05.141153 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" event={"ID":"9c7b2358-e2ca-4468-a192-c66914e53670","Type":"ContainerStarted","Data":"2f7eed48262f27bbf304e78474f98fbbf08a42cd71f0990af454c137a649981c"} Nov 01 15:47:05.141615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:05.141206 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" event={"ID":"9c7b2358-e2ca-4468-a192-c66914e53670","Type":"ContainerStarted","Data":"8bc00ed68446db5a25d2f5f551179f9e7a976ffbc0c5652ee60c651e38f05416"} Nov 01 15:47:05.257826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:05.257775 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" podStartSLOduration=2.257761829 podStartE2EDuration="2.257761829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:05.255070325 +0000 UTC m=+5364.313222175" watchObservedRunningTime="2026-11-01 15:47:05.257761829 +0000 UTC m=+5364.315913680" Nov 01 15:47:06.147857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:06.147824 5473 generic.go:356] "Generic (PLEG): container finished" podID="9c7b2358-e2ca-4468-a192-c66914e53670" containerID="2f7eed48262f27bbf304e78474f98fbbf08a42cd71f0990af454c137a649981c" exitCode=0 Nov 01 15:47:06.148300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:06.147889 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" event={"ID":"9c7b2358-e2ca-4468-a192-c66914e53670","Type":"ContainerDied","Data":"2f7eed48262f27bbf304e78474f98fbbf08a42cd71f0990af454c137a649981c"} Nov 01 15:47:07.289826 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.289800 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:07.351390 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.351369 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca\") pod \"9c7b2358-e2ca-4468-a192-c66914e53670\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " Nov 01 15:47:07.351495 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.351455 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v\" (UniqueName: \"kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v\") pod \"9c7b2358-e2ca-4468-a192-c66914e53670\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " Nov 01 15:47:07.351547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.351518 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca\") pod \"9c7b2358-e2ca-4468-a192-c66914e53670\" (UID: \"9c7b2358-e2ca-4468-a192-c66914e53670\") " Nov 01 15:47:07.351888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.351856 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca" pod "9c7b2358-e2ca-4468-a192-c66914e53670" (UID: "9c7b2358-e2ca-4468-a192-c66914e53670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:47:07.353507 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.353484 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca" pod "9c7b2358-e2ca-4468-a192-c66914e53670" (UID: "9c7b2358-e2ca-4468-a192-c66914e53670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:47:07.353584 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.353509 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v" pod "9c7b2358-e2ca-4468-a192-c66914e53670" (UID: "9c7b2358-e2ca-4468-a192-c66914e53670"). InnerVolumeSpecName "kube-api-access-pxd9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:47:07.453159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.453087 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:07.453159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.453116 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxd9v\" (UniqueName: \"kubernetes.io/projected/9c7b2358-e2ca-4468-a192-c66914e53670-kube-api-access-pxd9v\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:07.453159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:07.453127 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7b2358-e2ca-4468-a192-c66914e53670-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:08.160782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:08.160755 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" Nov 01 15:47:08.160962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:08.160753 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn" event={"ID":"9c7b2358-e2ca-4468-a192-c66914e53670","Type":"ContainerDied","Data":"8bc00ed68446db5a25d2f5f551179f9e7a976ffbc0c5652ee60c651e38f05416"} Nov 01 15:47:08.160962 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:08.160869 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bc00ed68446db5a25d2f5f551179f9e7a976ffbc0c5652ee60c651e38f05416" Nov 01 15:47:11.586850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:11.586816 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:22.660975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.660939 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg"] Nov 01 15:47:22.661397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.661356 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9c7b2358-e2ca-4468-a192-c66914e53670" containerName="revoke-keys" Nov 01 15:47:22.661397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.661367 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7b2358-e2ca-4468-a192-c66914e53670" containerName="revoke-keys" Nov 01 15:47:22.661482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.661441 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7b2358-e2ca-4468-a192-c66914e53670" containerName="revoke-keys" Nov 01 15:47:22.664881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.664865 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.744445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.744416 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.744609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.744454 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmxff\" (UniqueName: \"kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.744609 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.744473 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.767393 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.767354 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:47:22.767547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.767426 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:47:22.770115 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.770092 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg"] Nov 01 15:47:22.845753 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.845728 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmxff\" (UniqueName: \"kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.845917 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.845766 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.845999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.845919 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.846570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.846547 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.848005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.847984 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.902302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.902275 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmxff\" (UniqueName: \"kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff\") pod \"maas-api-revoke-sub-0165d7d2-2jwgg\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:22.978111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:22.978058 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:23.347396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:23.347367 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg"] Nov 01 15:47:23.348304 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:47:23.348275 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2407dc0_5119_4e7c_b808_3d0d39ec65b7.slice/crio-870b1e415f135359cf795afe99e1cbfe62c539affecc97c4d2c56d835145724d WatchSource:0}: Error finding container 870b1e415f135359cf795afe99e1cbfe62c539affecc97c4d2c56d835145724d: Status 404 returned error can't find the container with id 870b1e415f135359cf795afe99e1cbfe62c539affecc97c4d2c56d835145724d Nov 01 15:47:24.277067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:24.277031 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2407dc0-5119-4e7c-b808-3d0d39ec65b7" containerID="56753ffebbc4fd9dd9ea342c5c703b14a21a5c37e98f017f7cd659af6346ce03" exitCode=0 Nov 01 15:47:24.277476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:24.277084 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" event={"ID":"b2407dc0-5119-4e7c-b808-3d0d39ec65b7","Type":"ContainerDied","Data":"56753ffebbc4fd9dd9ea342c5c703b14a21a5c37e98f017f7cd659af6346ce03"} Nov 01 15:47:24.277476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:24.277136 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" event={"ID":"b2407dc0-5119-4e7c-b808-3d0d39ec65b7","Type":"ContainerStarted","Data":"870b1e415f135359cf795afe99e1cbfe62c539affecc97c4d2c56d835145724d"} Nov 01 15:47:25.420765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.420739 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:25.572846 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.572762 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff\" (UniqueName: \"kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff\") pod \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " Nov 01 15:47:25.573018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.572901 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca\") pod \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " Nov 01 15:47:25.573018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.573010 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca\") pod \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\" (UID: \"b2407dc0-5119-4e7c-b808-3d0d39ec65b7\") " Nov 01 15:47:25.573711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.573682 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca" pod "b2407dc0-5119-4e7c-b808-3d0d39ec65b7" (UID: "b2407dc0-5119-4e7c-b808-3d0d39ec65b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:47:25.576706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.576675 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca" pod "b2407dc0-5119-4e7c-b808-3d0d39ec65b7" (UID: "b2407dc0-5119-4e7c-b808-3d0d39ec65b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:47:25.576830 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.576729 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff" pod "b2407dc0-5119-4e7c-b808-3d0d39ec65b7" (UID: "b2407dc0-5119-4e7c-b808-3d0d39ec65b7"). InnerVolumeSpecName "kube-api-access-xmxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:47:25.674822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.674794 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:25.674959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.674838 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:25.674959 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:25.674849 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmxff\" (UniqueName: \"kubernetes.io/projected/b2407dc0-5119-4e7c-b808-3d0d39ec65b7-kube-api-access-xmxff\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:26.289349 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:26.289307 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" event={"ID":"b2407dc0-5119-4e7c-b808-3d0d39ec65b7","Type":"ContainerDied","Data":"870b1e415f135359cf795afe99e1cbfe62c539affecc97c4d2c56d835145724d"} Nov 01 15:47:26.289510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:26.289361 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg" Nov 01 15:47:26.289510 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:26.289371 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="870b1e415f135359cf795afe99e1cbfe62c539affecc97c4d2c56d835145724d" Nov 01 15:47:31.105734 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:47:31.105696 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:47:34.341890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.341860 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468"] Nov 01 15:47:34.342373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.342356 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2407dc0-5119-4e7c-b808-3d0d39ec65b7" containerName="revoke-keys" Nov 01 15:47:34.342431 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.342374 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2407dc0-5119-4e7c-b808-3d0d39ec65b7" containerName="revoke-keys" Nov 01 15:47:34.342473 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.342446 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2407dc0-5119-4e7c-b808-3d0d39ec65b7" containerName="revoke-keys" Nov 01 15:47:34.345717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.345703 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.395174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.395143 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468"] Nov 01 15:47:34.398126 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.398093 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:47:34.398299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.398198 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:47:34.454851 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.454818 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.454973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.454888 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.455034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.455009 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.555747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.555711 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.555905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.555780 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.555905 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.555867 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.556615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.556585 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.558189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.558169 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.612871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.612819 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf\") pod \"maas-api-revoke-sub-6b19338f-5t468\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:34.658795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:34.658773 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:35.031913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:35.031881 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468"] Nov 01 15:47:35.033273 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:47:35.033247 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf247ce16_b756_4465_a583_a881fd25c25c.slice/crio-4e255fb904d792124aa4fe92c754d3be41c09b03c74a24c15b4102bdedab4caa WatchSource:0}: Error finding container 4e255fb904d792124aa4fe92c754d3be41c09b03c74a24c15b4102bdedab4caa: Status 404 returned error can't find the container with id 4e255fb904d792124aa4fe92c754d3be41c09b03c74a24c15b4102bdedab4caa Nov 01 15:47:35.341802 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:35.341773 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" event={"ID":"f247ce16-b756-4465-a583-a881fd25c25c","Type":"ContainerStarted","Data":"4e255fb904d792124aa4fe92c754d3be41c09b03c74a24c15b4102bdedab4caa"} Nov 01 15:47:36.349643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:36.349605 5473 generic.go:356] "Generic (PLEG): container finished" podID="f247ce16-b756-4465-a583-a881fd25c25c" containerID="778d277983bfcdf9e3d16d603732f3317648da6183919d827b82cc1dfa7b9e87" exitCode=0 Nov 01 15:47:36.350036 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:36.349684 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" event={"ID":"f247ce16-b756-4465-a583-a881fd25c25c","Type":"ContainerDied","Data":"778d277983bfcdf9e3d16d603732f3317648da6183919d827b82cc1dfa7b9e87"} Nov 01 15:47:37.485971 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.485944 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:37.582840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.582806 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf\") pod \"f247ce16-b756-4465-a583-a881fd25c25c\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " Nov 01 15:47:37.583007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.582928 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca\") pod \"f247ce16-b756-4465-a583-a881fd25c25c\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " Nov 01 15:47:37.583007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.582961 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca\") pod \"f247ce16-b756-4465-a583-a881fd25c25c\" (UID: \"f247ce16-b756-4465-a583-a881fd25c25c\") " Nov 01 15:47:37.583293 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.583269 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca" pod "f247ce16-b756-4465-a583-a881fd25c25c" (UID: "f247ce16-b756-4465-a583-a881fd25c25c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:47:37.584924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.584889 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca" pod "f247ce16-b756-4465-a583-a881fd25c25c" (UID: "f247ce16-b756-4465-a583-a881fd25c25c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:47:37.585042 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.584987 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf" pod "f247ce16-b756-4465-a583-a881fd25c25c" (UID: "f247ce16-b756-4465-a583-a881fd25c25c"). InnerVolumeSpecName "kube-api-access-cwzrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:47:37.683760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.683687 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:37.683760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.683716 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwzrf\" (UniqueName: \"kubernetes.io/projected/f247ce16-b756-4465-a583-a881fd25c25c-kube-api-access-cwzrf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:37.683760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:37.683727 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f247ce16-b756-4465-a583-a881fd25c25c-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:47:38.362111 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:38.362079 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" Nov 01 15:47:38.362313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:38.362115 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468" event={"ID":"f247ce16-b756-4465-a583-a881fd25c25c","Type":"ContainerDied","Data":"4e255fb904d792124aa4fe92c754d3be41c09b03c74a24c15b4102bdedab4caa"} Nov 01 15:47:38.362313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:38.362161 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e255fb904d792124aa4fe92c754d3be41c09b03c74a24c15b4102bdedab4caa" Nov 01 15:47:44.759383 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:44.759357 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:47:44.786182 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:44.786158 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:47:45.259669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:45.259635 5473 scope.go:122] "RemoveContainer" containerID="779bbdf74079601aac6e51f973aa1fb8efe567ccf11ad481fcb7b7dfc270bf2e" Nov 01 15:47:45.268648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:45.268630 5473 scope.go:122] "RemoveContainer" containerID="f251bc2cee24b0c7d6d48dc5072fc3845b6d78cafbb1460be6a32fd9036563c1" Nov 01 15:47:49.634476 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:49.634428 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:50.231945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.231892 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9"] Nov 01 15:47:50.232541 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.232517 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f247ce16-b756-4465-a583-a881fd25c25c" containerName="revoke-keys" Nov 01 15:47:50.232672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.232543 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f247ce16-b756-4465-a583-a881fd25c25c" containerName="revoke-keys" Nov 01 15:47:50.232746 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.232694 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f247ce16-b756-4465-a583-a881fd25c25c" containerName="revoke-keys" Nov 01 15:47:50.237870 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.237846 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.322557 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.322483 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4dbaa3d3-epp-sa-dockercfg-vxrq6\"" Nov 01 15:47:50.322557 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.322491 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4dbaa3d3-kserve-self-signed-certs\"" Nov 01 15:47:50.328019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.327999 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs\") pod \"e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.328134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.328115 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbv7\" (UniqueName: \"kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7\") pod \"e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.342978 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.342954 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9"] Nov 01 15:47:50.429456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.429425 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs\") pod \"e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.429640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.429512 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbv7\" (UniqueName: \"kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7\") pod \"e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.431987 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.431961 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs\") pod \"e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.486806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.486749 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbv7\" (UniqueName: \"kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7\") pod \"e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.551980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.551938 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:50.924206 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.924178 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9"] Nov 01 15:47:50.925702 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:47:50.925661 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09f6e60_f43c_4dff_b3dc_eed386781881.slice/crio-20bdd57c17a3836ea0505e398dea30910f81fe61088d26c94fcf2afa654fcaa3 WatchSource:0}: Error finding container 20bdd57c17a3836ea0505e398dea30910f81fe61088d26c94fcf2afa654fcaa3: Status 404 returned error can't find the container with id 20bdd57c17a3836ea0505e398dea30910f81fe61088d26c94fcf2afa654fcaa3 Nov 01 15:47:50.927804 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:50.927784 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:47:51.167271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:51.167238 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:51.467102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:51.467070 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" event={"ID":"d09f6e60-f43c-4dff-b3dc-eed386781881","Type":"ContainerStarted","Data":"20bdd57c17a3836ea0505e398dea30910f81fe61088d26c94fcf2afa654fcaa3"} Nov 01 15:47:53.481016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:53.480926 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" event={"ID":"d09f6e60-f43c-4dff-b3dc-eed386781881","Type":"ContainerStarted","Data":"7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85"} Nov 01 15:47:53.481016 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:53.480987 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:47:53.595245 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:47:53.595180 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" podStartSLOduration=1.366233391 podStartE2EDuration="3.59516599s" totalImagesPullingTime="2.228932599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:53.593774223 +0000 UTC m=+5412.651926072" watchObservedRunningTime="2026-11-01 15:47:53.59516599 +0000 UTC m=+5412.653317896" Nov 01 15:48:01.106714 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:48:01.106681 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:48:09.216809 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:09.216773 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:11.892159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:11.892119 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:16.047939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:16.047886 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk"] Nov 01 15:48:16.050556 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:16.050527 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c79895f-jftpk"] Nov 01 15:48:17.080799 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:17.080768 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b520751d-61b0-4410-9d09-18c3148a422f" path="/var/lib/kubelet/pods/b520751d-61b0-4410-9d09-18c3148a422f/volumes" Nov 01 15:48:24.046899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:24.046867 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk"] Nov 01 15:48:24.096701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:24.096673 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9343056-5bwsk"] Nov 01 15:48:24.491724 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:24.491650 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:48:25.079751 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:25.079720 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6281a8be-7835-47d1-8254-e43fc2f9559f" path="/var/lib/kubelet/pods/6281a8be-7835-47d1-8254-e43fc2f9559f/volumes" Nov 01 15:48:28.048906 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:28.048873 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh"] Nov 01 15:48:28.098668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:28.098640 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6854c759-gkzhh"] Nov 01 15:48:29.079941 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:29.079909 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45353db-a197-4986-875c-2f4e39564e28" path="/var/lib/kubelet/pods/f45353db-a197-4986-875c-2f4e39564e28/volumes" Nov 01 15:48:31.106458 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:48:31.106418 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:48:36.049175 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:36.049140 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt"] Nov 01 15:48:36.055413 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:36.055388 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-537770ad-pk8gt"] Nov 01 15:48:37.079848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:37.079816 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d908adfd-9c4b-4bcc-a702-7e466d7fcf94" path="/var/lib/kubelet/pods/d908adfd-9c4b-4bcc-a702-7e466d7fcf94/volumes" Nov 01 15:48:58.707879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:58.707845 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9"] Nov 01 15:48:58.746924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:58.746893 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:58.917994 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:48:58.917965 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:01.106802 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:49:01.106760 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:49:04.491619 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:04.491577 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:49:12.226852 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:12.226810 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:12.384478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:12.384449 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:13.728828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:13.728769 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" containerID="cri-o://7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85" gracePeriod=45 Nov 01 15:49:14.491108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:14.491069 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:49:16.587027 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:16.586991 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:17.198009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:17.197963 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:17.397743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:17.397700 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:21.852650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:21.852576 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:22.023701 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:22.023672 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:22.198967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:22.198891 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:24.490796 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:24.490760 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:49:24.491196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:24.490872 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:49:24.924118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:24.924079 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:25.084411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:25.084378 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:27.459747 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:27.459712 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:27.616643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:27.616590 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:27.777679 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:27.777650 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:31.106403 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:49:31.106366 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:49:32.002290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:32.002257 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:34.491498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:34.491462 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 15:49:36.525961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:36.525928 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:41.020396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:41.020357 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:43.985513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:43.985482 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:49:44.068485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.068450 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7\" (UniqueName: \"kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7\") pod \"d09f6e60-f43c-4dff-b3dc-eed386781881\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " Nov 01 15:49:44.068696 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.068523 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs\") pod \"d09f6e60-f43c-4dff-b3dc-eed386781881\" (UID: \"d09f6e60-f43c-4dff-b3dc-eed386781881\") " Nov 01 15:49:44.071553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.071518 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7" pod "d09f6e60-f43c-4dff-b3dc-eed386781881" (UID: "d09f6e60-f43c-4dff-b3dc-eed386781881"). InnerVolumeSpecName "kube-api-access-szbv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:49:44.071720 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.071695 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs" pod "d09f6e60-f43c-4dff-b3dc-eed386781881" (UID: "d09f6e60-f43c-4dff-b3dc-eed386781881"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:49:44.170539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.170503 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbv7\" (UniqueName: \"kubernetes.io/projected/d09f6e60-f43c-4dff-b3dc-eed386781881-kube-api-access-szbv7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:44.170539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.170534 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d09f6e60-f43c-4dff-b3dc-eed386781881-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:44.248658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.248626 5473 generic.go:356] "Generic (PLEG): container finished" podID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerID="7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85" exitCode=0 Nov 01 15:49:44.248831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.248680 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" Nov 01 15:49:44.248831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.248715 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" event={"ID":"d09f6e60-f43c-4dff-b3dc-eed386781881","Type":"ContainerDied","Data":"7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85"} Nov 01 15:49:44.248831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.248771 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9" event={"ID":"d09f6e60-f43c-4dff-b3dc-eed386781881","Type":"ContainerDied","Data":"20bdd57c17a3836ea0505e398dea30910f81fe61088d26c94fcf2afa654fcaa3"} Nov 01 15:49:44.248831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.248794 5473 scope.go:122] "RemoveContainer" containerID="7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85" Nov 01 15:49:44.258735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.258716 5473 scope.go:122] "RemoveContainer" containerID="7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85" Nov 01 15:49:44.259010 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:49:44.258992 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85\": container with ID starting with 7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85 not found: ID does not exist" containerID="7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85" Nov 01 15:49:44.259086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.259017 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85"} err="failed to get container status \"7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85\": rpc error: code = NotFound desc = could not find container \"7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85\": container with ID starting with 7b3aa312ec1caca37ba1ec1823043729d72f1f780fcaed4e106f29d912ef0f85 not found: ID does not exist" Nov 01 15:49:44.412743 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.412668 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9"] Nov 01 15:49:44.416497 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:44.416474 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4dbaa3d3-kserve-router-scheduler-5498bf5bb6-w8qn9"] Nov 01 15:49:45.080517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:45.080476 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" path="/var/lib/kubelet/pods/d09f6e60-f43c-4dff-b3dc-eed386781881/volumes" Nov 01 15:49:45.307355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:45.307325 5473 scope.go:122] "RemoveContainer" containerID="b30e7b2ef6230b40fcaf84f6e656e4c91e2ef4bed3e4da6e9b570bfbb884f443" Nov 01 15:49:45.316302 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:45.316274 5473 scope.go:122] "RemoveContainer" containerID="d49f3d1f56197c8fd578437d3326089bb064fb75fe029aa7d76dbb1823a6dc6b" Nov 01 15:49:45.324396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:45.324380 5473 scope.go:122] "RemoveContainer" containerID="53cdb2316545a0bc3b06c9acd986db99a017f9109327a0151c891ebcb90b0699" Nov 01 15:49:45.332048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:45.332028 5473 scope.go:122] "RemoveContainer" containerID="8060f27a668e709a3afd10b8dcd46f44b1da2567c1161110927dcfeccf69b947" Nov 01 15:49:45.918135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:45.918106 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:46.662284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:46.662247 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:46.815838 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:46.815807 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:47.287155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:47.287123 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:47.451939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:47.451904 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:50.330897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.330864 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp"] Nov 01 15:49:50.331353 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.331335 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" Nov 01 15:49:50.331409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.331355 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" Nov 01 15:49:50.331461 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.331439 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09f6e60-f43c-4dff-b3dc-eed386781881" containerName="main" Nov 01 15:49:50.332865 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.332840 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:50.336597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.336578 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.432767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.432732 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.432767 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.432769 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kkz\" (UniqueName: \"kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.433046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.432848 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.433046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.432893 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.433046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.432973 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.433046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.433025 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.439752 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.439716 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9c2ba6\"/\"kube-root-ca.crt\"" Nov 01 15:49:50.439893 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.439768 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9c2ba6\"/\"openshift-service-ca.crt\"" Nov 01 15:49:50.439972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.439926 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9c2ba6\"/\"default-dockercfg-l85zc\"" Nov 01 15:49:50.440235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.440208 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9c2ba6\"/\"simulator-single-main-9c2ba6-kserve-self-signed-certs\"" Nov 01 15:49:50.442136 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.442112 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp"] Nov 01 15:49:50.501848 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.501819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:50.533945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.533918 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.533945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.533953 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kkz\" (UniqueName: \"kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.533991 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.534033 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.534051 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534261 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.534111 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534533 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.534507 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.534538 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.534599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.534556 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.536455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.536433 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.536903 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.536879 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.590409 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.590349 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kkz\" (UniqueName: \"kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz\") pod \"simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:50.653151 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:50.653127 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:51.023478 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:51.023449 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp"] Nov 01 15:49:51.025112 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:49:51.025082 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb76b45b_2a38_4352_863d_9afd16369d5e.slice/crio-b3925d2b16dceb60ff0609c0fe86f9efa1fb483ea79f6779a0479834288da1a3 WatchSource:0}: Error finding container b3925d2b16dceb60ff0609c0fe86f9efa1fb483ea79f6779a0479834288da1a3: Status 404 returned error can't find the container with id b3925d2b16dceb60ff0609c0fe86f9efa1fb483ea79f6779a0479834288da1a3 Nov 01 15:49:51.290046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:51.289959 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" event={"ID":"db76b45b-2a38-4352-863d-9afd16369d5e","Type":"ContainerStarted","Data":"9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74"} Nov 01 15:49:51.290046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:51.290007 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" event={"ID":"db76b45b-2a38-4352-863d-9afd16369d5e","Type":"ContainerStarted","Data":"b3925d2b16dceb60ff0609c0fe86f9efa1fb483ea79f6779a0479834288da1a3"} Nov 01 15:49:51.406181 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:51.406133 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" podStartSLOduration=1.406119244 podStartE2EDuration="1.406119244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:51.403608293 +0000 UTC m=+5530.461760143" watchObservedRunningTime="2026-11-01 15:49:51.406119244 +0000 UTC m=+5530.464271095" Nov 01 15:49:51.653305 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:51.653229 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:49:54.790419 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:54.790369 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:59.190519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:49:59.190487 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:00.653894 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:00.653858 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:50:00.666338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:00.666311 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:50:01.106947 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:50:01.106911 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:50:01.354271 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:01.354238 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:50:03.605936 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:03.605904 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:07.937299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:07.937262 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:08.093238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:08.093192 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:12.400996 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:12.400947 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:12.565850 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:12.565819 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:13.282307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:13.282276 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:13.440735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:13.440696 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:16.877594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:16.877563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:21.330866 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:21.330756 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:24.872153 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:24.872120 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:25.795415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:25.795383 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:30.208007 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:30.207968 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:31.106155 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:50:31.106123 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:50:32.987281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:32.987244 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc"] Nov 01 15:50:32.990878 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:32.990858 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.047945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.047919 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc"] Nov 01 15:50:33.049588 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.049562 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.049729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.049684 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpcqs\" (UniqueName: \"kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.049800 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.049761 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.094834 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.094807 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:50:33.094954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.094871 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:50:33.150550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.150521 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.150659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.150572 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpcqs\" (UniqueName: \"kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.150659 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.150609 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.151254 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.151232 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.152856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.152830 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.207290 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.207267 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpcqs\" (UniqueName: \"kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs\") pod \"maas-api-revoke-sub-78cbc85f-s8sbc\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.304140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.304112 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:33.673196 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:33.673163 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc"] Nov 01 15:50:33.674419 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:50:33.674391 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66458b29_874b_4084_88f3_ecf4991bc459.slice/crio-9ca714c94797715c2d4b291945286ff276d43d4453009afa4ceeeed3e961397f WatchSource:0}: Error finding container 9ca714c94797715c2d4b291945286ff276d43d4453009afa4ceeeed3e961397f: Status 404 returned error can't find the container with id 9ca714c94797715c2d4b291945286ff276d43d4453009afa4ceeeed3e961397f Nov 01 15:50:34.526876 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:34.526844 5473 generic.go:356] "Generic (PLEG): container finished" podID="66458b29-874b-4084-88f3-ecf4991bc459" containerID="cf3fcf2853ad3961b2168d52ea65fac29ee51d1486186e36bec609949ed48fd6" exitCode=0 Nov 01 15:50:34.527244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:34.526943 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" event={"ID":"66458b29-874b-4084-88f3-ecf4991bc459","Type":"ContainerDied","Data":"cf3fcf2853ad3961b2168d52ea65fac29ee51d1486186e36bec609949ed48fd6"} Nov 01 15:50:34.527244 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:34.526990 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" event={"ID":"66458b29-874b-4084-88f3-ecf4991bc459","Type":"ContainerStarted","Data":"9ca714c94797715c2d4b291945286ff276d43d4453009afa4ceeeed3e961397f"} Nov 01 15:50:34.590178 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:34.590153 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:35.665637 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.665614 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:35.776764 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.776732 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs\" (UniqueName: \"kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs\") pod \"66458b29-874b-4084-88f3-ecf4991bc459\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " Nov 01 15:50:35.776918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.776828 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca\") pod \"66458b29-874b-4084-88f3-ecf4991bc459\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " Nov 01 15:50:35.776918 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.776860 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca\") pod \"66458b29-874b-4084-88f3-ecf4991bc459\" (UID: \"66458b29-874b-4084-88f3-ecf4991bc459\") " Nov 01 15:50:35.777270 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.777245 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca" pod "66458b29-874b-4084-88f3-ecf4991bc459" (UID: "66458b29-874b-4084-88f3-ecf4991bc459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:50:35.779093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.779064 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca" pod "66458b29-874b-4084-88f3-ecf4991bc459" (UID: "66458b29-874b-4084-88f3-ecf4991bc459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:50:35.779334 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.779195 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs" pod "66458b29-874b-4084-88f3-ecf4991bc459" (UID: "66458b29-874b-4084-88f3-ecf4991bc459"). InnerVolumeSpecName "kube-api-access-dpcqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:50:35.878321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.878292 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpcqs\" (UniqueName: \"kubernetes.io/projected/66458b29-874b-4084-88f3-ecf4991bc459-kube-api-access-dpcqs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:35.878321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.878319 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:35.878512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:35.878330 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66458b29-874b-4084-88f3-ecf4991bc459-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:50:36.539784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:36.539752 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" Nov 01 15:50:36.539977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:36.539793 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc" event={"ID":"66458b29-874b-4084-88f3-ecf4991bc459","Type":"ContainerDied","Data":"9ca714c94797715c2d4b291945286ff276d43d4453009afa4ceeeed3e961397f"} Nov 01 15:50:36.539977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:36.539850 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca714c94797715c2d4b291945286ff276d43d4453009afa4ceeeed3e961397f" Nov 01 15:50:39.089669 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:39.089637 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:43.497895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:43.497859 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:44.075798 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:44.075748 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:44.553847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:44.553808 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:47.046819 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:47.046783 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72"] Nov 01 15:50:47.096475 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:47.096443 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mtm72"] Nov 01 15:50:47.910742 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:47.910711 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:49.008100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:49.008057 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp"] Nov 01 15:50:49.080156 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:49.080126 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062214c0-62a5-4fcb-bd46-22e0a21ddbb7" path="/var/lib/kubelet/pods/062214c0-62a5-4fcb-bd46-22e0a21ddbb7/volumes" Nov 01 15:50:49.102233 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:49.102193 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:49.931492 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:49.931456 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:50.544815 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:50.544713 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:50.922505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:50.922421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:51.083524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:51.083497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:51.493562 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:51.493523 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:51.655518 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:51.655483 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:52.429397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:52.429367 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:56.159339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:56.159304 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:56.561186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:50:56.561150 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:01.106867 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:51:01.106830 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:51:01.677033 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:01.676999 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:01.836294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:01.836261 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:04.028139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.028102 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" podUID="db76b45b-2a38-4352-863d-9afd16369d5e" containerName="main" containerID="cri-o://9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74" gracePeriod=45 Nov 01 15:51:04.284235 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.284157 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:51:04.361260 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361211 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache\") pod \"db76b45b-2a38-4352-863d-9afd16369d5e\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " Nov 01 15:51:04.361451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361292 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs\") pod \"db76b45b-2a38-4352-863d-9afd16369d5e\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " Nov 01 15:51:04.361451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361317 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir\") pod \"db76b45b-2a38-4352-863d-9afd16369d5e\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " Nov 01 15:51:04.361451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361394 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home\") pod \"db76b45b-2a38-4352-863d-9afd16369d5e\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " Nov 01 15:51:04.361451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361444 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz\" (UniqueName: \"kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz\") pod \"db76b45b-2a38-4352-863d-9afd16369d5e\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " Nov 01 15:51:04.361717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361508 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm\") pod \"db76b45b-2a38-4352-863d-9afd16369d5e\" (UID: \"db76b45b-2a38-4352-863d-9afd16369d5e\") " Nov 01 15:51:04.361717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361539 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache" pod "db76b45b-2a38-4352-863d-9afd16369d5e" (UID: "db76b45b-2a38-4352-863d-9afd16369d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:51:04.361717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361646 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir" pod "db76b45b-2a38-4352-863d-9afd16369d5e" (UID: "db76b45b-2a38-4352-863d-9afd16369d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:51:04.361717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361693 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home" pod "db76b45b-2a38-4352-863d-9afd16369d5e" (UID: "db76b45b-2a38-4352-863d-9afd16369d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:51:04.361970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361850 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:51:04.361970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361869 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:51:04.361970 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.361881 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:51:04.363676 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.363651 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm" pod "db76b45b-2a38-4352-863d-9afd16369d5e" (UID: "db76b45b-2a38-4352-863d-9afd16369d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:51:04.363784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.363679 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs" pod "db76b45b-2a38-4352-863d-9afd16369d5e" (UID: "db76b45b-2a38-4352-863d-9afd16369d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:51:04.363784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.363755 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz" pod "db76b45b-2a38-4352-863d-9afd16369d5e" (UID: "db76b45b-2a38-4352-863d-9afd16369d5e"). InnerVolumeSpecName "kube-api-access-b9kkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:51:04.462591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.462554 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9kkz\" (UniqueName: \"kubernetes.io/projected/db76b45b-2a38-4352-863d-9afd16369d5e-kube-api-access-b9kkz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:51:04.462591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.462588 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db76b45b-2a38-4352-863d-9afd16369d5e-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:51:04.462591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.462599 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db76b45b-2a38-4352-863d-9afd16369d5e-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:51:04.697402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.697319 5473 generic.go:356] "Generic (PLEG): container finished" podID="db76b45b-2a38-4352-863d-9afd16369d5e" containerID="9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74" exitCode=0 Nov 01 15:51:04.697402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.697372 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" Nov 01 15:51:04.697704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.697402 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" event={"ID":"db76b45b-2a38-4352-863d-9afd16369d5e","Type":"ContainerDied","Data":"9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74"} Nov 01 15:51:04.697704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.697445 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp" event={"ID":"db76b45b-2a38-4352-863d-9afd16369d5e","Type":"ContainerDied","Data":"b3925d2b16dceb60ff0609c0fe86f9efa1fb483ea79f6779a0479834288da1a3"} Nov 01 15:51:04.697704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.697461 5473 scope.go:122] "RemoveContainer" containerID="9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74" Nov 01 15:51:04.707015 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.706996 5473 scope.go:122] "RemoveContainer" containerID="9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74" Nov 01 15:51:04.707310 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:51:04.707290 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74\": container with ID starting with 9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74 not found: ID does not exist" containerID="9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74" Nov 01 15:51:04.707379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.707319 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74"} err="failed to get container status \"9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74\": rpc error: code = NotFound desc = could not find container \"9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74\": container with ID starting with 9ac9aa34b8ed1d4b897cc3435c28ed947cd8f5dcbbf45772fb0363cfd8d5ca74 not found: ID does not exist" Nov 01 15:51:04.861748 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.861717 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp"] Nov 01 15:51:04.865139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:04.865116 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9c2ba6/simulator-single-main-9c2ba6-kserve-6b88c649bf-l9gsp"] Nov 01 15:51:05.081100 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:05.081063 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db76b45b-2a38-4352-863d-9afd16369d5e" path="/var/lib/kubelet/pods/db76b45b-2a38-4352-863d-9afd16369d5e/volumes" Nov 01 15:51:12.927924 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:12.927890 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:25.512898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:25.512852 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:26.124775 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:26.124736 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:28.248186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:28.248148 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:28.422269 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:28.422211 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:29.260879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:29.260848 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:29.505821 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:29.505788 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:30.131980 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:30.131952 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:31.106326 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:51:31.106294 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:51:31.675994 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:31.675957 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:31.869378 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:31.869342 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:35.050553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:35.050504 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m"] Nov 01 15:51:35.100320 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:35.100287 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lgm4m"] Nov 01 15:51:37.080412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:37.080375 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f6605c-9daf-4027-871d-2b216a0c42fb" path="/var/lib/kubelet/pods/95f6605c-9daf-4027-871d-2b216a0c42fb/volumes" Nov 01 15:51:45.380101 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:45.380065 5473 scope.go:122] "RemoveContainer" containerID="1412a24b1097e427a3460c5cbb491dfd05faf4437d251ead5aba9d1e72ce1aa1" Nov 01 15:51:45.389058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:45.389035 5473 scope.go:122] "RemoveContainer" containerID="80aff74cd0c2d5b04d147bacb8219f962d3f6ce5fdd85d6992f243ce6221ba2b" Nov 01 15:51:46.074097 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074063 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z"] Nov 01 15:51:46.074498 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074483 5473 cpu_manager.go:414] "containerMap: removing container" podUID="db76b45b-2a38-4352-863d-9afd16369d5e" containerName="main" Nov 01 15:51:46.074555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074500 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db76b45b-2a38-4352-863d-9afd16369d5e" containerName="main" Nov 01 15:51:46.074555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074514 5473 cpu_manager.go:414] "containerMap: removing container" podUID="66458b29-874b-4084-88f3-ecf4991bc459" containerName="revoke-keys" Nov 01 15:51:46.074555 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074518 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66458b29-874b-4084-88f3-ecf4991bc459" containerName="revoke-keys" Nov 01 15:51:46.074684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074599 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="db76b45b-2a38-4352-863d-9afd16369d5e" containerName="main" Nov 01 15:51:46.074684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.074608 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="66458b29-874b-4084-88f3-ecf4991bc459" containerName="revoke-keys" Nov 01 15:51:46.079275 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.079252 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.133937 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.133901 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z"] Nov 01 15:51:46.166957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.166926 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.167114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.166962 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.167114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.167057 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.167114 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.167089 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2sbr\" (UniqueName: \"kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.167288 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.167261 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.167342 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.167330 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.181202 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.181177 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-835e7bd2-kserve-self-signed-certs\"" Nov 01 15:51:46.224105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.224073 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:46.268272 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268243 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268284 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268331 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268350 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268392 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268453 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268418 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2sbr\" (UniqueName: \"kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268658 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268779 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268754 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.268871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.268795 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.270602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.270580 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.270920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.270901 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.325324 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.325272 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2sbr\" (UniqueName: \"kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr\") pod \"e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.391030 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.391007 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:46.768512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.768487 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z"] Nov 01 15:51:46.770282 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:51:46.770252 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc91ae586_e683_48ab_9222_80edec6e482b.slice/crio-f17fc246d128dc707a683523aa1785f63d309ccc3028f2f62b24d3decf3ec046 WatchSource:0}: Error finding container f17fc246d128dc707a683523aa1785f63d309ccc3028f2f62b24d3decf3ec046: Status 404 returned error can't find the container with id f17fc246d128dc707a683523aa1785f63d309ccc3028f2f62b24d3decf3ec046 Nov 01 15:51:46.944119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.944072 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" event={"ID":"c91ae586-e683-48ab-9222-80edec6e482b","Type":"ContainerStarted","Data":"6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf"} Nov 01 15:51:46.944321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:46.944126 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" event={"ID":"c91ae586-e683-48ab-9222-80edec6e482b","Type":"ContainerStarted","Data":"f17fc246d128dc707a683523aa1785f63d309ccc3028f2f62b24d3decf3ec046"} Nov 01 15:51:47.058373 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:47.058266 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" podStartSLOduration=1.058251163 podStartE2EDuration="1.058251163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:51:47.057375464 +0000 UTC m=+5646.115527313" watchObservedRunningTime="2026-11-01 15:51:47.058251163 +0000 UTC m=+5646.116403013" Nov 01 15:51:47.391772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:47.391689 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:56.391496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:56.391447 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:56.404321 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:56.404296 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:51:57.030856 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:51:57.030816 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:52:01.107638 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:52:01.107599 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:52:08.046684 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:08.046649 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn"] Nov 01 15:52:08.096508 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:08.096471 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-qpltn"] Nov 01 15:52:09.080397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:09.080364 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7b2358-e2ca-4468-a192-c66914e53670" path="/var/lib/kubelet/pods/9c7b2358-e2ca-4468-a192-c66914e53670/volumes" Nov 01 15:52:13.174759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:13.174725 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:15.554504 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:15.554464 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:26.046972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:26.046884 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg"] Nov 01 15:52:26.095434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:26.095400 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-2jwgg"] Nov 01 15:52:27.080676 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:27.080641 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2407dc0-5119-4e7c-b808-3d0d39ec65b7" path="/var/lib/kubelet/pods/b2407dc0-5119-4e7c-b808-3d0d39ec65b7/volumes" Nov 01 15:52:31.106170 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:52:31.106135 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:52:38.049143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.049103 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468"] Nov 01 15:52:38.098947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.098921 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-5t468"] Nov 01 15:52:38.143845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.143802 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl"] Nov 01 15:52:38.150843 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.150812 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.201780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.201748 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl"] Nov 01 15:52:38.203183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.203149 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:52:38.203338 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.203309 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:52:38.242999 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.242972 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.243134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.243021 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hgd\" (UniqueName: \"kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.243134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.243094 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.344466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.344391 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.344628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.344518 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.344628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.344575 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hgd\" (UniqueName: \"kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.345065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.345045 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.346795 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.346774 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.401343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.401319 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hgd\" (UniqueName: \"kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd\") pod \"maas-api-revoke-sub-452b6d7b-5hbpl\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.464379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.464354 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:38.833543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:38.833516 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl"] Nov 01 15:52:38.835827 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:52:38.835800 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca42330_9af9_4ba9_80ad_5cc47880102c.slice/crio-5c4439e32e180981149dd4a0dfdc62eab8762539ed778cd8c25d6160fcd31681 WatchSource:0}: Error finding container 5c4439e32e180981149dd4a0dfdc62eab8762539ed778cd8c25d6160fcd31681: Status 404 returned error can't find the container with id 5c4439e32e180981149dd4a0dfdc62eab8762539ed778cd8c25d6160fcd31681 Nov 01 15:52:39.080294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:39.080263 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f247ce16-b756-4465-a583-a881fd25c25c" path="/var/lib/kubelet/pods/f247ce16-b756-4465-a583-a881fd25c25c/volumes" Nov 01 15:52:39.276647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:39.276607 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" event={"ID":"fca42330-9af9-4ba9-80ad-5cc47880102c","Type":"ContainerStarted","Data":"cc2c8e340fbaaf22ece06b775d23c5c9ec7d7452424c5bcdec8153aac5c9b44f"} Nov 01 15:52:39.276831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:39.276665 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" event={"ID":"fca42330-9af9-4ba9-80ad-5cc47880102c","Type":"ContainerStarted","Data":"5c4439e32e180981149dd4a0dfdc62eab8762539ed778cd8c25d6160fcd31681"} Nov 01 15:52:39.390823 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:39.390730 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" podStartSLOduration=1.39071634 podStartE2EDuration="1.39071634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:39.38901969 +0000 UTC m=+5698.447171540" watchObservedRunningTime="2026-11-01 15:52:39.39071634 +0000 UTC m=+5698.448868183" Nov 01 15:52:40.286551 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:40.286509 5473 generic.go:356] "Generic (PLEG): container finished" podID="fca42330-9af9-4ba9-80ad-5cc47880102c" containerID="cc2c8e340fbaaf22ece06b775d23c5c9ec7d7452424c5bcdec8153aac5c9b44f" exitCode=0 Nov 01 15:52:40.287099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:40.286575 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" event={"ID":"fca42330-9af9-4ba9-80ad-5cc47880102c","Type":"ContainerDied","Data":"cc2c8e340fbaaf22ece06b775d23c5c9ec7d7452424c5bcdec8153aac5c9b44f"} Nov 01 15:52:42.263242 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.263029 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:42.336319 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.336272 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca\") pod \"fca42330-9af9-4ba9-80ad-5cc47880102c\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " Nov 01 15:52:42.337332 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.337310 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca\") pod \"fca42330-9af9-4ba9-80ad-5cc47880102c\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " Nov 01 15:52:42.338142 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.338120 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca" pod "fca42330-9af9-4ba9-80ad-5cc47880102c" (UID: "fca42330-9af9-4ba9-80ad-5cc47880102c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:52:42.338323 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.338285 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd\" (UniqueName: \"kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd\") pod \"fca42330-9af9-4ba9-80ad-5cc47880102c\" (UID: \"fca42330-9af9-4ba9-80ad-5cc47880102c\") " Nov 01 15:52:42.340675 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.340656 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:42.364839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.364543 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" event={"ID":"fca42330-9af9-4ba9-80ad-5cc47880102c","Type":"ContainerDied","Data":"5c4439e32e180981149dd4a0dfdc62eab8762539ed778cd8c25d6160fcd31681"} Nov 01 15:52:42.364839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.364599 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c4439e32e180981149dd4a0dfdc62eab8762539ed778cd8c25d6160fcd31681" Nov 01 15:52:42.364839 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.364668 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl" Nov 01 15:52:42.365782 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.365760 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca" pod "fca42330-9af9-4ba9-80ad-5cc47880102c" (UID: "fca42330-9af9-4ba9-80ad-5cc47880102c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:52:42.373059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.373023 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd" pod "fca42330-9af9-4ba9-80ad-5cc47880102c" (UID: "fca42330-9af9-4ba9-80ad-5cc47880102c"). InnerVolumeSpecName "kube-api-access-t5hgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:42.475739 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.475651 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fca42330-9af9-4ba9-80ad-5cc47880102c-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:42.476000 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.475964 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hgd\" (UniqueName: \"kubernetes.io/projected/fca42330-9af9-4ba9-80ad-5cc47880102c-kube-api-access-t5hgd\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:42.579397 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.579341 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:42.579692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.579649 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:42.601369 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.601327 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:42.601612 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.601586 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:42.644334 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.644302 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:42.644565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.644538 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:42.807772 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.807740 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:42.808093 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:42.808057 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:44.799406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:44.799376 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:52:44.832168 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:44.832146 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:52:45.419759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:45.419726 5473 scope.go:122] "RemoveContainer" containerID="469b3d12fe1a8d2ac7ea8747954bbb42995d4e1b48f0f06013181e71bc96cc41" Nov 01 15:52:50.840847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:50.840817 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z"] Nov 01 15:52:50.881841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:50.881808 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:53.614933 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:53.614902 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:57.580183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.580126 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:57.582012 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.581966 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:57.601359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.601323 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:57.601642 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.601614 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:57.643945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.643905 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:57.644284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.644249 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:57.810402 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.810354 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:52:57.810744 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:57.810695 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:52:58.442117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.442081 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c"] Nov 01 15:52:58.442550 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.442536 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fca42330-9af9-4ba9-80ad-5cc47880102c" containerName="revoke-keys" Nov 01 15:52:58.442600 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.442551 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca42330-9af9-4ba9-80ad-5cc47880102c" containerName="revoke-keys" Nov 01 15:52:58.442647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.442627 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca42330-9af9-4ba9-80ad-5cc47880102c" containerName="revoke-keys" Nov 01 15:52:58.445681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.445664 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.478279 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.478243 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.478418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.478315 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.478418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.478375 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptbbm\" (UniqueName: \"kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.503022 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.502986 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c"] Nov 01 15:52:58.550287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.550252 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:52:58.550287 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.550269 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:52:58.579713 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.579678 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptbbm\" (UniqueName: \"kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.579858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.579788 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.579858 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.579828 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.580512 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.580494 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.582240 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.582206 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.636913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.636882 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptbbm\" (UniqueName: \"kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm\") pod \"maas-api-revoke-sub-3e31a506-twg7c\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:58.758953 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:58.758872 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:52:59.129294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:59.129267 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c"] Nov 01 15:52:59.130847 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:52:59.130818 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa15ae82_a2a8_4138_8cdb_ebdf5467f101.slice/crio-efc77cf009400ec6961c3d1b27ca95783809e785aecf6a9f8ca9471b114cbc89 WatchSource:0}: Error finding container efc77cf009400ec6961c3d1b27ca95783809e785aecf6a9f8ca9471b114cbc89: Status 404 returned error can't find the container with id efc77cf009400ec6961c3d1b27ca95783809e785aecf6a9f8ca9471b114cbc89 Nov 01 15:52:59.492183 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:59.492147 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" event={"ID":"fa15ae82-a2a8-4138-8cdb-ebdf5467f101","Type":"ContainerStarted","Data":"6aa6ab20f9bd938aa2486a4be8189bc86aa49460fed82b4c542a042723498d92"} Nov 01 15:52:59.492345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:59.492209 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" event={"ID":"fa15ae82-a2a8-4138-8cdb-ebdf5467f101","Type":"ContainerStarted","Data":"efc77cf009400ec6961c3d1b27ca95783809e785aecf6a9f8ca9471b114cbc89"} Nov 01 15:52:59.606662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:52:59.606608 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" podStartSLOduration=1.606594079 podStartE2EDuration="1.606594079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:59.604595743 +0000 UTC m=+5718.662747585" watchObservedRunningTime="2026-11-01 15:52:59.606594079 +0000 UTC m=+5718.664745929" Nov 01 15:53:00.498717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:00.498677 5473 generic.go:356] "Generic (PLEG): container finished" podID="fa15ae82-a2a8-4138-8cdb-ebdf5467f101" containerID="6aa6ab20f9bd938aa2486a4be8189bc86aa49460fed82b4c542a042723498d92" exitCode=0 Nov 01 15:53:00.498717 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:00.498710 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" event={"ID":"fa15ae82-a2a8-4138-8cdb-ebdf5467f101","Type":"ContainerDied","Data":"6aa6ab20f9bd938aa2486a4be8189bc86aa49460fed82b4c542a042723498d92"} Nov 01 15:53:01.110115 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:53:01.110063 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:53:01.716913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.716874 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:53:01.818370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.818254 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca\") pod \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " Nov 01 15:53:01.818370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.818340 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm\" (UniqueName: \"kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm\") pod \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " Nov 01 15:53:01.824429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.824387 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca\") pod \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\" (UID: \"fa15ae82-a2a8-4138-8cdb-ebdf5467f101\") " Nov 01 15:53:01.825135 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.819196 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca" pod "fa15ae82-a2a8-4138-8cdb-ebdf5467f101" (UID: "fa15ae82-a2a8-4138-8cdb-ebdf5467f101"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:53:01.831304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.829758 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca" pod "fa15ae82-a2a8-4138-8cdb-ebdf5467f101" (UID: "fa15ae82-a2a8-4138-8cdb-ebdf5467f101"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:53:01.831304 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.829858 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm" pod "fa15ae82-a2a8-4138-8cdb-ebdf5467f101" (UID: "fa15ae82-a2a8-4138-8cdb-ebdf5467f101"). InnerVolumeSpecName "kube-api-access-ptbbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:53:01.926406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.926359 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:01.926406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.926401 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:01.926406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:01.926417 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptbbm\" (UniqueName: \"kubernetes.io/projected/fa15ae82-a2a8-4138-8cdb-ebdf5467f101-kube-api-access-ptbbm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:02.528771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:02.528733 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" event={"ID":"fa15ae82-a2a8-4138-8cdb-ebdf5467f101","Type":"ContainerDied","Data":"efc77cf009400ec6961c3d1b27ca95783809e785aecf6a9f8ca9471b114cbc89"} Nov 01 15:53:02.529433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:02.529411 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc77cf009400ec6961c3d1b27ca95783809e785aecf6a9f8ca9471b114cbc89" Nov 01 15:53:02.529651 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:02.529637 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c" Nov 01 15:53:05.860530 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:05.860493 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" podUID="c91ae586-e683-48ab-9222-80edec6e482b" containerName="main" containerID="cri-o://6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf" gracePeriod=45 Nov 01 15:53:06.020119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.020073 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" podUID="c91ae586-e683-48ab-9222-80edec6e482b" containerName="main" probeResult="failure" output="Get \"https://10.134.1.118:8000/ready\": dial tcp 10.134.1.118:8000: connect: connection refused" Nov 01 15:53:06.108912 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.108883 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:53:06.220119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220031 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home\") pod \"c91ae586-e683-48ab-9222-80edec6e482b\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " Nov 01 15:53:06.220119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220069 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs\") pod \"c91ae586-e683-48ab-9222-80edec6e482b\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " Nov 01 15:53:06.220119 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220121 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache\") pod \"c91ae586-e683-48ab-9222-80edec6e482b\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " Nov 01 15:53:06.220440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220183 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr\" (UniqueName: \"kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr\") pod \"c91ae586-e683-48ab-9222-80edec6e482b\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " Nov 01 15:53:06.220440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220249 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir\") pod \"c91ae586-e683-48ab-9222-80edec6e482b\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " Nov 01 15:53:06.220440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220305 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm\") pod \"c91ae586-e683-48ab-9222-80edec6e482b\" (UID: \"c91ae586-e683-48ab-9222-80edec6e482b\") " Nov 01 15:53:06.220440 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220386 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home" pod "c91ae586-e683-48ab-9222-80edec6e482b" (UID: "c91ae586-e683-48ab-9222-80edec6e482b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:53:06.220667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220455 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache" pod "c91ae586-e683-48ab-9222-80edec6e482b" (UID: "c91ae586-e683-48ab-9222-80edec6e482b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:53:06.220667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220546 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir" pod "c91ae586-e683-48ab-9222-80edec6e482b" (UID: "c91ae586-e683-48ab-9222-80edec6e482b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:53:06.220792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220746 5473 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-home\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:06.220792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220768 5473 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-model-cache\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:06.220792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.220782 5473 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-tmp-dir\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:06.222543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.222517 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm" pod "c91ae586-e683-48ab-9222-80edec6e482b" (UID: "c91ae586-e683-48ab-9222-80edec6e482b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:53:06.223340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.223288 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr" pod "c91ae586-e683-48ab-9222-80edec6e482b" (UID: "c91ae586-e683-48ab-9222-80edec6e482b"). InnerVolumeSpecName "kube-api-access-t2sbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:53:06.223561 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.223536 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs" pod "c91ae586-e683-48ab-9222-80edec6e482b" (UID: "c91ae586-e683-48ab-9222-80edec6e482b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:53:06.322134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.322095 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2sbr\" (UniqueName: \"kubernetes.io/projected/c91ae586-e683-48ab-9222-80edec6e482b-kube-api-access-t2sbr\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:06.322134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.322133 5473 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91ae586-e683-48ab-9222-80edec6e482b-dshm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:06.322363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.322145 5473 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91ae586-e683-48ab-9222-80edec6e482b-tls-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:06.567429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.567395 5473 generic.go:356] "Generic (PLEG): container finished" podID="c91ae586-e683-48ab-9222-80edec6e482b" containerID="6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf" exitCode=0 Nov 01 15:53:06.567631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.567438 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" Nov 01 15:53:06.567631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.567476 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" event={"ID":"c91ae586-e683-48ab-9222-80edec6e482b","Type":"ContainerDied","Data":"6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf"} Nov 01 15:53:06.567631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.567520 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z" event={"ID":"c91ae586-e683-48ab-9222-80edec6e482b","Type":"ContainerDied","Data":"f17fc246d128dc707a683523aa1785f63d309ccc3028f2f62b24d3decf3ec046"} Nov 01 15:53:06.567631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.567537 5473 scope.go:122] "RemoveContainer" containerID="6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf" Nov 01 15:53:06.579496 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.579458 5473 scope.go:122] "RemoveContainer" containerID="6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf" Nov 01 15:53:06.579859 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:53:06.579830 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf\": container with ID starting with 6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf not found: ID does not exist" containerID="6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf" Nov 01 15:53:06.579934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.579871 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf"} err="failed to get container status \"6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf\": rpc error: code = NotFound desc = could not find container \"6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf\": container with ID starting with 6fd1d55228df9fc4b306e59395e317e5b1416d49c92644028611a4f19e37dfcf not found: ID does not exist" Nov 01 15:53:06.731294 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.731253 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z"] Nov 01 15:53:06.780787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:06.780750 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-835e7bd2-kserve-56dd44f46f-qws8z"] Nov 01 15:53:07.080139 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:07.080105 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c91ae586-e683-48ab-9222-80edec6e482b" path="/var/lib/kubelet/pods/c91ae586-e683-48ab-9222-80edec6e482b/volumes" Nov 01 15:53:13.608973 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:13.608831 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:22.924416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924385 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9"] Nov 01 15:53:22.924808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924796 5473 cpu_manager.go:414] "containerMap: removing container" podUID="fa15ae82-a2a8-4138-8cdb-ebdf5467f101" containerName="revoke-keys" Nov 01 15:53:22.924857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924809 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa15ae82-a2a8-4138-8cdb-ebdf5467f101" containerName="revoke-keys" Nov 01 15:53:22.924857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924828 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c91ae586-e683-48ab-9222-80edec6e482b" containerName="main" Nov 01 15:53:22.924857 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924833 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c91ae586-e683-48ab-9222-80edec6e482b" containerName="main" Nov 01 15:53:22.924967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924903 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa15ae82-a2a8-4138-8cdb-ebdf5467f101" containerName="revoke-keys" Nov 01 15:53:22.924967 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.924913 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c91ae586-e683-48ab-9222-80edec6e482b" containerName="main" Nov 01 15:53:22.928164 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.928139 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:22.998165 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.998127 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:22.998377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.998307 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:22.998377 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:22.998367 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwcg\" (UniqueName: \"kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.032703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.032668 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:53:23.032703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.032681 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:53:23.034945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.034924 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9"] Nov 01 15:53:23.099682 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.099645 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.099888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.099717 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.099888 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.099745 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwcg\" (UniqueName: \"kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.100433 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.100406 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.102048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.102029 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.156198 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.156165 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwcg\" (UniqueName: \"kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg\") pod \"maas-api-revoke-sub-81566d09-46vd9\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.242662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.242566 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:23.612463 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.612436 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9"] Nov 01 15:53:23.614677 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:53:23.614647 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b73c6a_5cef_4d8a_a3b5_1b8636774e1e.slice/crio-0cc2eebfbfaaefd7ef14fbc81320bd0a7c4ac200a8c083145cf9a6855773068b WatchSource:0}: Error finding container 0cc2eebfbfaaefd7ef14fbc81320bd0a7c4ac200a8c083145cf9a6855773068b: Status 404 returned error can't find the container with id 0cc2eebfbfaaefd7ef14fbc81320bd0a7c4ac200a8c083145cf9a6855773068b Nov 01 15:53:23.762740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:23.762703 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" event={"ID":"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e","Type":"ContainerStarted","Data":"0cc2eebfbfaaefd7ef14fbc81320bd0a7c4ac200a8c083145cf9a6855773068b"} Nov 01 15:53:24.770611 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:24.770570 5473 generic.go:356] "Generic (PLEG): container finished" podID="52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" containerID="b32c67f11b4cc8d1c65e4e4a790834119b29e06a42d589b618939011c0a486df" exitCode=0 Nov 01 15:53:24.771006 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:24.770657 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" event={"ID":"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e","Type":"ContainerDied","Data":"b32c67f11b4cc8d1c65e4e4a790834119b29e06a42d589b618939011c0a486df"} Nov 01 15:53:25.914125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.914097 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:25.925285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.925256 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg\" (UniqueName: \"kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg\") pod \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " Nov 01 15:53:25.925472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.925368 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca\") pod \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " Nov 01 15:53:25.925472 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.925465 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca\") pod \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\" (UID: \"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e\") " Nov 01 15:53:25.925998 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.925963 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca" pod "52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" (UID: "52b73c6a-5cef-4d8a-a3b5-1b8636774e1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:53:25.927529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.927506 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca" pod "52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" (UID: "52b73c6a-5cef-4d8a-a3b5-1b8636774e1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:53:25.927711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:25.927691 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg" pod "52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" (UID: "52b73c6a-5cef-4d8a-a3b5-1b8636774e1e"). InnerVolumeSpecName "kube-api-access-9zwcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:53:26.027123 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:26.027035 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:26.027359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:26.027297 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:26.027359 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:26.027322 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zwcg\" (UniqueName: \"kubernetes.io/projected/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e-kube-api-access-9zwcg\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:26.783085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:26.783049 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" Nov 01 15:53:26.783285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:26.783082 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9" event={"ID":"52b73c6a-5cef-4d8a-a3b5-1b8636774e1e","Type":"ContainerDied","Data":"0cc2eebfbfaaefd7ef14fbc81320bd0a7c4ac200a8c083145cf9a6855773068b"} Nov 01 15:53:26.783285 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:26.783131 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cc2eebfbfaaefd7ef14fbc81320bd0a7c4ac200a8c083145cf9a6855773068b" Nov 01 15:53:31.106567 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:53:31.106527 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:53:34.986995 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:34.986959 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:35.589493 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:35.589457 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:35.799623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:35.799586 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:40.216791 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:40.216754 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:40.379085 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:40.379052 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:40.614355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:40.614320 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:43.274853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.274817 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:43.444831 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.444796 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:43.648262 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.648160 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9"] Nov 01 15:53:43.648598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.648585 5473 cpu_manager.go:414] "containerMap: removing container" podUID="52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" containerName="revoke-keys" Nov 01 15:53:43.648648 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.648599 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" containerName="revoke-keys" Nov 01 15:53:43.648698 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.648689 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" containerName="revoke-keys" Nov 01 15:53:43.651920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.651902 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.702086 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.702052 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume\") pod \"payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.702301 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.702096 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhnx\" (UniqueName: \"kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx\") pod \"payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.710811 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.710780 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9"] Nov 01 15:53:43.757192 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.757160 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-dual-3436ba\"" Nov 01 15:53:43.757392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.757367 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-dual-3436ba-dockercfg-swwqk\"" Nov 01 15:53:43.802895 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.802860 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhnx\" (UniqueName: \"kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx\") pod \"payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.803082 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.802997 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume\") pod \"payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.803801 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.803776 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume\") pod \"payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.812674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.812647 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st"] Nov 01 15:53:43.816250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.816205 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:43.860299 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.860268 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhnx\" (UniqueName: \"kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx\") pod \"payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:43.872687 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.872642 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st"] Nov 01 15:53:43.904166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.904087 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume\") pod \"payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:43.904166 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.904128 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6gs\" (UniqueName: \"kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs\") pod \"payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:43.964771 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:43.964737 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:44.005282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.005240 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume\") pod \"payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:44.005488 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.005306 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6gs\" (UniqueName: \"kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs\") pod \"payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:44.006159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.006135 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume\") pod \"payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:44.062249 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.062186 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6gs\" (UniqueName: \"kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs\") pod \"payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:44.129482 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.129449 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:44.337370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.337339 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9"] Nov 01 15:53:44.338422 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:53:44.338393 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1fd01c_e38b_4c01_86cf_cd7813dc176d.slice/crio-1c01f0b6c851e8ea6253cb199181707d3e1be09ea8c7bbb74561b89df5647918 WatchSource:0}: Error finding container 1c01f0b6c851e8ea6253cb199181707d3e1be09ea8c7bbb74561b89df5647918: Status 404 returned error can't find the container with id 1c01f0b6c851e8ea6253cb199181707d3e1be09ea8c7bbb74561b89df5647918 Nov 01 15:53:44.501131 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.501099 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st"] Nov 01 15:53:44.501976 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:53:44.501954 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4107ef_5ec4_45c4_8f0e_ed33f5ff1aca.slice/crio-2c2e08d4cc6fd375dad9fd2b24fff730ebde2c59b6070505bf2bed2a8e9f13e5 WatchSource:0}: Error finding container 2c2e08d4cc6fd375dad9fd2b24fff730ebde2c59b6070505bf2bed2a8e9f13e5: Status 404 returned error can't find the container with id 2c2e08d4cc6fd375dad9fd2b24fff730ebde2c59b6070505bf2bed2a8e9f13e5 Nov 01 15:53:44.885286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.885244 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" event={"ID":"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca","Type":"ContainerStarted","Data":"3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c"} Nov 01 15:53:44.885506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.885312 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:44.885506 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.885338 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" event={"ID":"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca","Type":"ContainerStarted","Data":"2c2e08d4cc6fd375dad9fd2b24fff730ebde2c59b6070505bf2bed2a8e9f13e5"} Nov 01 15:53:44.886915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.886890 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" event={"ID":"7c1fd01c-e38b-4c01-86cf-cd7813dc176d","Type":"ContainerStarted","Data":"d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11"} Nov 01 15:53:44.887039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.886937 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" event={"ID":"7c1fd01c-e38b-4c01-86cf-cd7813dc176d","Type":"ContainerStarted","Data":"1c01f0b6c851e8ea6253cb199181707d3e1be09ea8c7bbb74561b89df5647918"} Nov 01 15:53:44.887039 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:44.886970 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:45.001343 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:45.001280 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" podStartSLOduration=2.001260601 podStartE2EDuration="2.001260601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:44.998445787 +0000 UTC m=+5764.056597637" watchObservedRunningTime="2026-11-01 15:53:45.001260601 +0000 UTC m=+5764.059412456" Nov 01 15:53:45.443532 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:45.443500 5473 scope.go:122] "RemoveContainer" containerID="778d277983bfcdf9e3d16d603732f3317648da6183919d827b82cc1dfa7b9e87" Nov 01 15:53:45.452379 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:45.452358 5473 scope.go:122] "RemoveContainer" containerID="56753ffebbc4fd9dd9ea342c5c703b14a21a5c37e98f017f7cd659af6346ce03" Nov 01 15:53:45.460418 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:45.460393 5473 scope.go:122] "RemoveContainer" containerID="2f7eed48262f27bbf304e78474f98fbbf08a42cd71f0990af454c137a649981c" Nov 01 15:53:46.161125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:46.161088 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:46.319414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:46.319383 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:46.480358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:46.480271 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:50.702580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:50.702497 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:55.203414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:55.203382 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:55.896549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:55.896518 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:53:55.896777 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:55.896677 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:53:56.014144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:56.014091 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" podStartSLOduration=13.014077663 podStartE2EDuration="13.014077663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:45.120090298 +0000 UTC m=+5764.178242149" watchObservedRunningTime="2026-11-01 15:53:56.014077663 +0000 UTC m=+5775.072229512" Nov 01 15:53:59.702951 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:53:59.702914 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:00.246077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:00.246047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:01.107190 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:54:01.107156 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:54:04.495836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:04.495793 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:05.795585 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.795547 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv"] Nov 01 15:54:05.798992 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.798975 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821093 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821124 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821282 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821157 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821293 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821356 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821416 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821384 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821462 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821500 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.821674 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.821571 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fmsm\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.902054 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.902024 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-dual-3436ba-openshift-default-dockercfg-hdq2b\"" Nov 01 15:54:05.905558 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.905529 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv"] Nov 01 15:54:05.922487 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922451 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922494 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922540 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922569 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922726 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922766 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922803 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922848 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.922989 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.922898 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fmsm\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.923130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.923031 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.923130 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.923080 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.923292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.923139 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.923827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.923772 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.923945 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.923850 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.925389 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.925368 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.925513 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.925495 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.979322 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.979289 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:05.979756 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:05.979735 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fmsm\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-65bqv\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:06.112899 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:06.112793 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:06.539853 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:06.539817 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv"] Nov 01 15:54:06.540007 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:54:06.539981 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edaf3a6_d167_422a_b702_c93895a8f4da.slice/crio-2e4520c47a2d4a7985aaea7e4245ddbbee49d018fa68690f19658f54d083a7bd WatchSource:0}: Error finding container 2e4520c47a2d4a7985aaea7e4245ddbbee49d018fa68690f19658f54d083a7bd: Status 404 returned error can't find the container with id 2e4520c47a2d4a7985aaea7e4245ddbbee49d018fa68690f19658f54d083a7bd Nov 01 15:54:06.542449 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:06.542413 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:06.542574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:06.542476 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:06.542574 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:06.542508 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:07.012363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:07.012336 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" event={"ID":"5edaf3a6-d167-422a-b702-c93895a8f4da","Type":"ContainerStarted","Data":"acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454"} Nov 01 15:54:07.012787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:07.012383 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" event={"ID":"5edaf3a6-d167-422a-b702-c93895a8f4da","Type":"ContainerStarted","Data":"2e4520c47a2d4a7985aaea7e4245ddbbee49d018fa68690f19658f54d083a7bd"} Nov 01 15:54:07.113607 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:07.113569 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:07.114991 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:07.114963 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Nov 01 15:54:07.115068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:07.115017 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Nov 01 15:54:07.219599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:07.219541 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podStartSLOduration=2.219526089 podStartE2EDuration="2.219526089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:07.177300417 +0000 UTC m=+5786.235452268" watchObservedRunningTime="2026-11-01 15:54:07.219526089 +0000 UTC m=+5786.277677944" Nov 01 15:54:08.114118 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:08.114077 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Nov 01 15:54:08.114580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:08.114138 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Nov 01 15:54:09.054686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:09.054651 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:09.114184 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:09.114142 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Nov 01 15:54:09.114631 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:09.114247 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Nov 01 15:54:10.114059 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:10.114023 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Nov 01 15:54:10.114263 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:10.114074 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Nov 01 15:54:11.113307 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:11.113269 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Nov 01 15:54:11.113517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:11.113326 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Nov 01 15:54:12.113099 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:12.113062 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" start-of-body= Nov 01 15:54:12.113536 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:12.113114 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.124:15021/healthz/ready\": dial tcp 10.134.1.124:15021: connect: connection refused" Nov 01 15:54:13.118421 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:13.118375 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:13.118919 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:13.118892 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:13.119789 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:13.119765 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:54:13.706593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:13.706549 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:13.904281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:13.904200 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:18.157829 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:18.157801 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:22.504005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:22.503963 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:26.984658 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:26.984601 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:31.106397 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:54:31.106361 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:54:31.433066 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:31.432956 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:35.855891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:35.855854 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:36.036174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.036137 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm"] Nov 01 15:54:36.039765 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.039749 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.085088 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.085055 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf"] Nov 01 15:54:36.088706 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.088685 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.141177 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.141105 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm"] Nov 01 15:54:36.148090 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148057 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148131 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148266 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148251 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148294 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148376 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9k9k\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148409 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148427 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148447 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.148578 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.148541 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.192106 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.192060 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf"] Nov 01 15:54:36.249892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.249850 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.249892 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.249896 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.249926 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250022 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250059 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250089 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250120 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250164 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250197 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250243 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250293 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9k9k\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250331 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250357 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250393 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250421 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66h7\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250482 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250506 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.250516 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250519 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.251155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.250802 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.251155 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.251107 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.251316 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.251292 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.251385 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.251325 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.251519 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.251499 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.252120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.252097 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.253926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.253902 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.307458 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.307429 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.307640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.307526 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9k9k\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.352345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352305 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352365 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352387 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352426 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352449 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352523 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352489 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352628 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352727 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66h7\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.352845 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352783 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.353031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352904 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.353031 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.352978 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.353157 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.353117 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.353283 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.353261 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:36.353364 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.353261 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.353624 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.353591 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.354602 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.354583 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.355065 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.355043 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.410652 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.410623 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66h7\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.411403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.411376 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token\") pod \"e2e-dual-3436ba-openshift-default-8446b54f85-55bbf\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.701281 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.701172 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:36.778047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.777957 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm"] Nov 01 15:54:36.780033 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:54:36.779989 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479a69d0_7355_417a_a723_d09721c4c9d5.slice/crio-01079946300bbd5ddc28abc1606574127e5d50d3be3d5462ae3894c48139d32f WatchSource:0}: Error finding container 01079946300bbd5ddc28abc1606574127e5d50d3be3d5462ae3894c48139d32f: Status 404 returned error can't find the container with id 01079946300bbd5ddc28abc1606574127e5d50d3be3d5462ae3894c48139d32f Nov 01 15:54:36.782286 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.782254 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:36.782405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.782331 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:36.782405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:36.782378 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:37.077323 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:54:37.077291 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4de405_3a85_405f_a67c_a1055f955791.slice/crio-d9d4218e5445e0d78667f865cfe9c1d5c77ada86dcfb2d5306b6e2d3eb8c9bbc WatchSource:0}: Error finding container d9d4218e5445e0d78667f865cfe9c1d5c77ada86dcfb2d5306b6e2d3eb8c9bbc: Status 404 returned error can't find the container with id d9d4218e5445e0d78667f865cfe9c1d5c77ada86dcfb2d5306b6e2d3eb8c9bbc Nov 01 15:54:37.080102 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.080070 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:37.080238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.080132 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:37.080238 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.080162 5473 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 01 15:54:37.081681 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.081661 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf"] Nov 01 15:54:37.205068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.205027 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" event={"ID":"479a69d0-7355-417a-a723-d09721c4c9d5","Type":"ContainerStarted","Data":"29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c"} Nov 01 15:54:37.205308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.205088 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" event={"ID":"479a69d0-7355-417a-a723-d09721c4c9d5","Type":"ContainerStarted","Data":"01079946300bbd5ddc28abc1606574127e5d50d3be3d5462ae3894c48139d32f"} Nov 01 15:54:37.207686 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.207656 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" event={"ID":"ab4de405-3a85-405f-a67c-a1055f955791","Type":"ContainerStarted","Data":"7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7"} Nov 01 15:54:37.207833 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.207703 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" event={"ID":"ab4de405-3a85-405f-a67c-a1055f955791","Type":"ContainerStarted","Data":"d9d4218e5445e0d78667f865cfe9c1d5c77ada86dcfb2d5306b6e2d3eb8c9bbc"} Nov 01 15:54:37.355517 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.355406 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:37.357273 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.357244 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 01 15:54:37.357443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.357291 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 01 15:54:37.457242 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.457040 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podStartSLOduration=2.457018842 podStartE2EDuration="2.457018842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:37.338578688 +0000 UTC m=+5816.396730538" watchObservedRunningTime="2026-11-01 15:54:37.457018842 +0000 UTC m=+5816.515170693" Nov 01 15:54:37.701883 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:37.701797 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:38.354778 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:38.354735 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 01 15:54:38.355443 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:38.355396 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 01 15:54:38.702914 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:38.702832 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 15:54:38.703140 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:38.703119 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 15:54:39.357729 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:39.357688 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 01 15:54:39.358403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:39.358357 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 01 15:54:39.702456 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:39.702353 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 15:54:39.702672 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:39.702649 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 15:54:40.358078 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:40.358037 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 01 15:54:40.367703 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:40.367662 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 01 15:54:40.380370 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:40.377075 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:40.702103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:40.702012 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 15:54:40.702103 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:40.702064 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 15:54:41.356107 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:41.356068 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 01 15:54:41.356847 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:41.356818 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 01 15:54:41.702549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:41.702457 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 15:54:41.702549 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:41.702515 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 15:54:42.354630 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:42.354590 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 01 15:54:42.354954 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:42.354926 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 01 15:54:42.702610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:42.702526 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 01 15:54:42.702610 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:42.702579 5473 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 01 15:54:43.359147 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.359113 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:43.359629 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.359597 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:43.360545 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.360523 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:54:43.497719 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.497669 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podStartSLOduration=7.4976534 podStartE2EDuration="7.4976534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:37.459884089 +0000 UTC m=+5816.518035955" watchObservedRunningTime="2026-11-01 15:54:43.4976534 +0000 UTC m=+5822.555805250" Nov 01 15:54:43.706109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.706033 5473 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:43.706553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.706475 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:43.707075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:43.707058 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:54:44.731417 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:44.731380 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:49.195075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:49.195042 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:53.607524 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:53.607490 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:58.121723 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:54:58.121695 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:01.107260 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:55:01.107208 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:55:02.489521 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:02.489491 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:06.909040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:06.909004 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:11.304480 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:11.304444 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:14.225783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:14.225743 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:15.784722 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:15.784684 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:20.202670 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:20.202635 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:24.622803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:24.622772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:29.025911 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:29.025881 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:31.105957 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:55:31.105921 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:55:33.505236 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:33.505181 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:36.049491 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:36.049458 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc"] Nov 01 15:55:36.092117 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:36.092093 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78cbc85f-s8sbc"] Nov 01 15:55:37.080565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:37.080522 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66458b29-874b-4084-88f3-ecf4991bc459" path="/var/lib/kubelet/pods/66458b29-874b-4084-88f3-ecf4991bc459/volumes" Nov 01 15:55:37.898842 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:37.898810 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:42.300334 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:42.300299 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:46.712560 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:46.712516 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:51.188340 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:51.188302 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:54.367887 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:54.367853 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:55.594931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:55:55.594899 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:00.043455 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:00.043421 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:00.197414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:00.197379 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:00.787047 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:00.787009 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st"] Nov 01 15:56:00.886697 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:00.886666 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9"] Nov 01 15:56:01.110370 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:01.110291 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:56:01.162780 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:01.162744 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:01.317424 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:01.317390 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:04.711784 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:04.711748 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:05.624563 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:05.624528 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:05.787451 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:05.787411 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" podUID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" containerName="payload-processing" containerID="cri-o://3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c" gracePeriod=25 Nov 01 15:56:05.887201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:05.887128 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" podUID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" containerName="payload-pre-processing" containerID="cri-o://d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11" gracePeriod=25 Nov 01 15:56:05.894420 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:56:05.894398 5473 logging.go:55] [core] [Channel #22202 SubChannel #22204]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.123:50052", ServerName: "10.134.1.123:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.123:50052: connect: connection refused" Nov 01 15:56:05.894548 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:56:05.894405 5473 logging.go:55] [core] [Channel #22201 SubChannel #22203]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.122:50052", ServerName: "10.134.1.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.122:50052: connect: connection refused" Nov 01 15:56:06.042653 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.042622 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:56:06.064955 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.064923 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume\") pod \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " Nov 01 15:56:06.065120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.065071 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs\" (UniqueName: \"kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs\") pod \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\" (UID: \"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca\") " Nov 01 15:56:06.065426 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.065398 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume" pod "df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" (UID: "df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:56:06.080505 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.080474 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs" pod "df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" (UID: "df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca"). InnerVolumeSpecName "kube-api-access-sv6gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:06.138840 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.138777 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:56:06.166467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.166435 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx\" (UniqueName: \"kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx\") pod \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " Nov 01 15:56:06.166613 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.166560 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume\") pod \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\" (UID: \"7c1fd01c-e38b-4c01-86cf-cd7813dc176d\") " Nov 01 15:56:06.167011 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.166980 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume" pod "7c1fd01c-e38b-4c01-86cf-cd7813dc176d" (UID: "7c1fd01c-e38b-4c01-86cf-cd7813dc176d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:56:06.167149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.167012 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6gs\" (UniqueName: \"kubernetes.io/projected/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-kube-api-access-sv6gs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:06.167149 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.167031 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:06.168543 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.168522 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx" pod "7c1fd01c-e38b-4c01-86cf-cd7813dc176d" (UID: "7c1fd01c-e38b-4c01-86cf-cd7813dc176d"). InnerVolumeSpecName "kube-api-access-5nhnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:06.267750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.267718 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhnx\" (UniqueName: \"kubernetes.io/projected/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-kube-api-access-5nhnx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:06.267750 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.267747 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c1fd01c-e38b-4c01-86cf-cd7813dc176d-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:06.765138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.765105 5473 generic.go:356] "Generic (PLEG): container finished" podID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" containerID="3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c" exitCode=0 Nov 01 15:56:06.765355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.765166 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" Nov 01 15:56:06.765355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.765184 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" event={"ID":"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca","Type":"ContainerDied","Data":"3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c"} Nov 01 15:56:06.765355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.765268 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" event={"ID":"df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca","Type":"ContainerDied","Data":"2c2e08d4cc6fd375dad9fd2b24fff730ebde2c59b6070505bf2bed2a8e9f13e5"} Nov 01 15:56:06.765355 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.765292 5473 scope.go:122] "RemoveContainer" containerID="3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c" Nov 01 15:56:06.766787 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.766762 5473 generic.go:356] "Generic (PLEG): container finished" podID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" containerID="d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11" exitCode=0 Nov 01 15:56:06.766939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.766797 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" event={"ID":"7c1fd01c-e38b-4c01-86cf-cd7813dc176d","Type":"ContainerDied","Data":"d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11"} Nov 01 15:56:06.766939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.766807 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" Nov 01 15:56:06.766939 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.766837 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" event={"ID":"7c1fd01c-e38b-4c01-86cf-cd7813dc176d","Type":"ContainerDied","Data":"1c01f0b6c851e8ea6253cb199181707d3e1be09ea8c7bbb74561b89df5647918"} Nov 01 15:56:06.775256 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.775240 5473 scope.go:122] "RemoveContainer" containerID="3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c" Nov 01 15:56:06.775531 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:06.775509 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c\": container with ID starting with 3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c not found: ID does not exist" containerID="3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c" Nov 01 15:56:06.775627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.775536 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c"} err="failed to get container status \"3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c\": rpc error: code = NotFound desc = could not find container \"3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c\": container with ID starting with 3f37634b7fb197a71cf24578149f52becda9eca7bd72250323bb183b61c2fa1c not found: ID does not exist" Nov 01 15:56:06.775627 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.775556 5473 scope.go:122] "RemoveContainer" containerID="d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11" Nov 01 15:56:06.784352 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.784337 5473 scope.go:122] "RemoveContainer" containerID="d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11" Nov 01 15:56:06.784580 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:06.784559 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11\": container with ID starting with d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11 not found: ID does not exist" containerID="d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11" Nov 01 15:56:06.784643 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.784586 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11"} err="failed to get container status \"d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11\": rpc error: code = NotFound desc = could not find container \"d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11\": container with ID starting with d1fe5e31a4a76887c70df2e902a6a8897dc9bd034166462521d78c0eec239d11 not found: ID does not exist" Nov 01 15:56:06.894554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.894521 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9" podUID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.122:50052\" within 1s: context deadline exceeded" Nov 01 15:56:06.894554 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.894544 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st" podUID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.123:50052\" within 1s: context deadline exceeded" Nov 01 15:56:06.929134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.929086 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9"] Nov 01 15:56:06.932005 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:06.931979 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-dual-3436ba-77f9944fb7-bgfg9"] Nov 01 15:56:07.040442 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:07.040376 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st"] Nov 01 15:56:07.045582 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:07.045560 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-dual-3436ba-5dc4dbbf59-tn8st"] Nov 01 15:56:07.080745 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:07.080717 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" path="/var/lib/kubelet/pods/7c1fd01c-e38b-4c01-86cf-cd7813dc176d/volumes" Nov 01 15:56:07.081109 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:07.081096 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" path="/var/lib/kubelet/pods/df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca/volumes" Nov 01 15:56:08.577436 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:08.577401 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:11.219565 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:11.219536 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:11.375566 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:11.375474 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:12.683581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:12.683546 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:13.226490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.226455 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm"] Nov 01 15:56:13.226700 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.226657 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" containerID="cri-o://29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c" gracePeriod=30 Nov 01 15:56:13.242403 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.242373 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv"] Nov 01 15:56:13.242623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.242600 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" containerID="cri-o://acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454" gracePeriod=30 Nov 01 15:56:13.259068 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.259041 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:13.270485 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.270457 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf"] Nov 01 15:56:13.270657 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.270639 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" containerID="cri-o://7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7" gracePeriod=30 Nov 01 15:56:13.362380 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.362347 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:56:13.362559 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.362399 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:56:13.708292 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.708252 5473 patch_prober.go:29] interesting pod/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 15:56:13.708704 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:13.708313 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 15:56:14.547897 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:14.547867 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:18.530640 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.530615 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:56:18.563863 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.563844 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:56:18.567040 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.567020 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:56:18.592004 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.591976 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592025 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592069 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592088 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592125 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592105 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592150 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592255 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592306 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592411 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592336 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo\") pod \"5edaf3a6-d167-422a-b702-c93895a8f4da\" (UID: \"5edaf3a6-d167-422a-b702-c93895a8f4da\") " Nov 01 15:56:18.592647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592428 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.592647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592445 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.592647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592550 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.592647 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592603 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:56:18.592879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592669 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.592879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592852 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edaf3a6-d167-422a-b702-c93895a8f4da-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.592879 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592872 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.593063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592888 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.593063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592902 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.593063 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.592916 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.594358 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.594336 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.594459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.594354 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "kube-api-access-9fmsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:18.594459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.594387 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:56:18.594459 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.594440 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token" pod "5edaf3a6-d167-422a-b702-c93895a8f4da" (UID: "5edaf3a6-d167-422a-b702-c93895a8f4da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:18.635251 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635230 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5f8877f69-h4kjb"] Nov 01 15:56:18.635599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635588 5473 cpu_manager.go:414] "containerMap: removing container" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" Nov 01 15:56:18.635649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635600 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" Nov 01 15:56:18.635649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635616 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" Nov 01 15:56:18.635649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635621 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" Nov 01 15:56:18.635649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635642 5473 cpu_manager.go:414] "containerMap: removing container" podUID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" containerName="payload-pre-processing" Nov 01 15:56:18.635649 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635647 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" containerName="payload-pre-processing" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635657 5473 cpu_manager.go:414] "containerMap: removing container" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635662 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635672 5473 cpu_manager.go:414] "containerMap: removing container" podUID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" containerName="payload-processing" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635676 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" containerName="payload-processing" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635747 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerName="istio-proxy" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635759 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1fd01c-e38b-4c01-86cf-cd7813dc176d" containerName="payload-pre-processing" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635766 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" containerName="istio-proxy" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635773 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4107ef-5ec4-45c4-8f0e-ed33f5ff1aca" containerName="payload-processing" Nov 01 15:56:18.635816 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.635780 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4de405-3a85-405f-a67c-a1055f955791" containerName="istio-proxy" Nov 01 15:56:18.639019 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.639006 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.678499 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.678472 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5f8877f69-h4kjb"] Nov 01 15:56:18.691274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.691243 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 15:56:18.691414 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.691336 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-h5hgn\"" Nov 01 15:56:18.693693 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693670 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.693808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693710 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.693808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693741 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.693808 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693783 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.694001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693820 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.694001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693855 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.694001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693893 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693948 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.694001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693975 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.694001 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693983 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.693978 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694040 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694080 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694165 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694177 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694184 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694208 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694274 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694314 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694364 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.694392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694396 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket\") pod \"479a69d0-7355-417a-a723-d09721c4c9d5\" (UID: \"479a69d0-7355-417a-a723-d09721c4c9d5\") " Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694413 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694411 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694428 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token\") pod \"ab4de405-3a85-405f-a67c-a1055f955791\" (UID: \"ab4de405-3a85-405f-a67c-a1055f955791\") " Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694656 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume\") pod \"payload-pre-processing-5f8877f69-h4kjb\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694815 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv762\" (UniqueName: \"kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762\") pod \"payload-pre-processing-5f8877f69-h4kjb\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694913 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694931 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fmsm\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-kube-api-access-9fmsm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694947 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694948 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694962 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694978 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.694991 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695006 5473 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-data\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695018 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695020 5473 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-workload-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695036 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695051 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edaf3a6-d167-422a-b702-c93895a8f4da-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.695794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695178 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.695794 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695751 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.696035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.695943 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:56:18.696594 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.696572 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.696692 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.696635 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:56:18.696977 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.696954 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "kube-api-access-c66h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:18.697250 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.697205 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:56:18.697881 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.697857 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token" pod "ab4de405-3a85-405f-a67c-a1055f955791" (UID: "ab4de405-3a85-405f-a67c-a1055f955791"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:18.697972 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.697907 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "kube-api-access-n9k9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:18.698122 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.698099 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:56:18.698662 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.698642 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo" pod "479a69d0-7355-417a-a723-d09721c4c9d5" (UID: "479a69d0-7355-417a-a723-d09721c4c9d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 15:56:18.796412 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796383 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume\") pod \"payload-pre-processing-5f8877f69-h4kjb\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.796567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796473 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv762\" (UniqueName: \"kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762\") pod \"payload-pre-processing-5f8877f69-h4kjb\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.796567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796529 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66h7\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-kube-api-access-c66h7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796542 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796552 5473 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-istio-envoy\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796561 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab4de405-3a85-405f-a67c-a1055f955791-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796567 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796569 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/479a69d0-7355-417a-a723-d09721c4c9d5-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796578 5473 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab4de405-3a85-405f-a67c-a1055f955791-workload-certs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796586 5473 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab4de405-3a85-405f-a67c-a1055f955791-istiod-ca-cert\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796597 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9k9k\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-kube-api-access-n9k9k\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796611 5473 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/479a69d0-7355-417a-a723-d09721c4c9d5-istio-podinfo\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796619 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/479a69d0-7355-417a-a723-d09721c4c9d5-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796629 5473 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/479a69d0-7355-417a-a723-d09721c4c9d5-credential-socket\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.796932 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.796637 5473 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab4de405-3a85-405f-a67c-a1055f955791-istio-token\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:56:18.797174 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.797074 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume\") pod \"payload-pre-processing-5f8877f69-h4kjb\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.836625 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.836589 5473 generic.go:356] "Generic (PLEG): container finished" podID="5edaf3a6-d167-422a-b702-c93895a8f4da" containerID="acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454" exitCode=0 Nov 01 15:56:18.836806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.836655 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" Nov 01 15:56:18.836806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.836678 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" event={"ID":"5edaf3a6-d167-422a-b702-c93895a8f4da","Type":"ContainerDied","Data":"acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454"} Nov 01 15:56:18.836806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.836726 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv" event={"ID":"5edaf3a6-d167-422a-b702-c93895a8f4da","Type":"ContainerDied","Data":"2e4520c47a2d4a7985aaea7e4245ddbbee49d018fa68690f19658f54d083a7bd"} Nov 01 15:56:18.836806 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.836755 5473 scope.go:122] "RemoveContainer" containerID="acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454" Nov 01 15:56:18.838479 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.838452 5473 generic.go:356] "Generic (PLEG): container finished" podID="479a69d0-7355-417a-a723-d09721c4c9d5" containerID="29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c" exitCode=0 Nov 01 15:56:18.838634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.838485 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" Nov 01 15:56:18.838634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.838536 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" event={"ID":"479a69d0-7355-417a-a723-d09721c4c9d5","Type":"ContainerDied","Data":"29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c"} Nov 01 15:56:18.838634 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.838600 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm" event={"ID":"479a69d0-7355-417a-a723-d09721c4c9d5","Type":"ContainerDied","Data":"01079946300bbd5ddc28abc1606574127e5d50d3be3d5462ae3894c48139d32f"} Nov 01 15:56:18.840306 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.840280 5473 generic.go:356] "Generic (PLEG): container finished" podID="ab4de405-3a85-405f-a67c-a1055f955791" containerID="7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7" exitCode=0 Nov 01 15:56:18.840467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.840341 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" Nov 01 15:56:18.840467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.840345 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" event={"ID":"ab4de405-3a85-405f-a67c-a1055f955791","Type":"ContainerDied","Data":"7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7"} Nov 01 15:56:18.840467 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.840405 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf" event={"ID":"ab4de405-3a85-405f-a67c-a1055f955791","Type":"ContainerDied","Data":"d9d4218e5445e0d78667f865cfe9c1d5c77ada86dcfb2d5306b6e2d3eb8c9bbc"} Nov 01 15:56:18.854069 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.854047 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv762\" (UniqueName: \"kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762\") pod \"payload-pre-processing-5f8877f69-h4kjb\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:18.854190 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.854171 5473 scope.go:122] "RemoveContainer" containerID="acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454" Nov 01 15:56:18.854580 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:18.854541 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454\": container with ID starting with acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454 not found: ID does not exist" containerID="acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454" Nov 01 15:56:18.854580 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.854579 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454"} err="failed to get container status \"acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454\": rpc error: code = NotFound desc = could not find container \"acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454\": container with ID starting with acb1cb03857a4383260da63473cee701943acd2b76d227e6db1df90166a51454 not found: ID does not exist" Nov 01 15:56:18.854760 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.854601 5473 scope.go:122] "RemoveContainer" containerID="29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c" Nov 01 15:56:18.863080 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.863058 5473 scope.go:122] "RemoveContainer" containerID="29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c" Nov 01 15:56:18.863377 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:18.863335 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c\": container with ID starting with 29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c not found: ID does not exist" containerID="29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c" Nov 01 15:56:18.863457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.863387 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c"} err="failed to get container status \"29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c\": rpc error: code = NotFound desc = could not find container \"29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c\": container with ID starting with 29d5bf4bb4b29e49415dcd90b9749e12b9d1342f000bca71186388aae4fafe8c not found: ID does not exist" Nov 01 15:56:18.863457 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.863409 5473 scope.go:122] "RemoveContainer" containerID="7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7" Nov 01 15:56:18.872035 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.872016 5473 scope.go:122] "RemoveContainer" containerID="7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7" Nov 01 15:56:18.872318 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:18.872298 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7\": container with ID starting with 7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7 not found: ID does not exist" containerID="7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7" Nov 01 15:56:18.872375 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.872323 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7"} err="failed to get container status \"7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7\": rpc error: code = NotFound desc = could not find container \"7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7\": container with ID starting with 7407b8237c8fe94a5fb55682eca131f9fbc5fe6412bbab51c2f802699f0844e7 not found: ID does not exist" Nov 01 15:56:18.951327 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:18.951299 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:19.154144 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.154056 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf"] Nov 01 15:56:19.158073 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.158047 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-55bbf"] Nov 01 15:56:19.267400 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.267372 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm"] Nov 01 15:56:19.276553 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.276525 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-8gmgm"] Nov 01 15:56:19.328406 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.328381 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5f8877f69-h4kjb"] Nov 01 15:56:19.331037 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:56:19.331006 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a3d980_99f0_4771_a933_056cd7a9225c.slice/crio-41a84cc8686e2cae9a510addc205ef36f58b068872c3d9585bfa553328b60860 WatchSource:0}: Error finding container 41a84cc8686e2cae9a510addc205ef36f58b068872c3d9585bfa553328b60860: Status 404 returned error can't find the container with id 41a84cc8686e2cae9a510addc205ef36f58b068872c3d9585bfa553328b60860 Nov 01 15:56:19.386204 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.386177 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv"] Nov 01 15:56:19.391540 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.391518 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-dual-3436ba-openshift-default-8446b54f85-65bqv"] Nov 01 15:56:19.850855 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.850818 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" event={"ID":"78a3d980-99f0-4771-a933-056cd7a9225c","Type":"ContainerStarted","Data":"e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536"} Nov 01 15:56:19.851337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.850873 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" event={"ID":"78a3d980-99f0-4771-a933-056cd7a9225c","Type":"ContainerStarted","Data":"41a84cc8686e2cae9a510addc205ef36f58b068872c3d9585bfa553328b60860"} Nov 01 15:56:19.851337 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.850906 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:19.966734 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:19.966671 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" podStartSLOduration=1.9666521449999999 podStartE2EDuration="1.966652145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:19.963577169 +0000 UTC m=+5919.021729018" watchObservedRunningTime="2026-11-01 15:56:19.966652145 +0000 UTC m=+5919.024803998" Nov 01 15:56:21.081638 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:21.081603 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479a69d0-7355-417a-a723-d09721c4c9d5" path="/var/lib/kubelet/pods/479a69d0-7355-417a-a723-d09721c4c9d5/volumes" Nov 01 15:56:21.082052 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:21.082038 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edaf3a6-d167-422a-b702-c93895a8f4da" path="/var/lib/kubelet/pods/5edaf3a6-d167-422a-b702-c93895a8f4da/volumes" Nov 01 15:56:21.082471 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:21.082458 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4de405-3a85-405f-a67c-a1055f955791" path="/var/lib/kubelet/pods/ab4de405-3a85-405f-a67c-a1055f955791/volumes" Nov 01 15:56:30.861535 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:30.861508 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:56:31.108757 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:56:31.108712 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:56:45.509110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:56:45.509070 5473 scope.go:122] "RemoveContainer" containerID="cf3fcf2853ad3961b2168d52ea65fac29ee51d1486186e36bec609949ed48fd6" Nov 01 15:57:01.108016 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:57:01.107922 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:57:01.444898 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:01.444818 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5f8877f69-h4kjb"] Nov 01 15:57:01.745807 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:01.745732 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:01.901949 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:01.901916 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:03.826735 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:03.826699 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:04.250891 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:04.250817 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:06.445284 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.445246 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" podUID="78a3d980-99f0-4771-a933-056cd7a9225c" containerName="payload-pre-processing" containerID="cri-o://e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536" gracePeriod=25 Nov 01 15:57:06.685591 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.685564 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:06.707246 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.707181 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:57:06.854143 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.854106 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume\") pod \"78a3d980-99f0-4771-a933-056cd7a9225c\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " Nov 01 15:57:06.854425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.854186 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762\" (UniqueName: \"kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762\") pod \"78a3d980-99f0-4771-a933-056cd7a9225c\" (UID: \"78a3d980-99f0-4771-a933-056cd7a9225c\") " Nov 01 15:57:06.854529 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.854505 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume" pod "78a3d980-99f0-4771-a933-056cd7a9225c" (UID: "78a3d980-99f0-4771-a933-056cd7a9225c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:57:06.856408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.856385 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762" pod "78a3d980-99f0-4771-a933-056cd7a9225c" (UID: "78a3d980-99f0-4771-a933-056cd7a9225c"). InnerVolumeSpecName "kube-api-access-rv762". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:57:06.955189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.955161 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv762\" (UniqueName: \"kubernetes.io/projected/78a3d980-99f0-4771-a933-056cd7a9225c-kube-api-access-rv762\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:57:06.955189 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:06.955188 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78a3d980-99f0-4771-a933-056cd7a9225c-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:57:07.128034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.127988 5473 generic.go:356] "Generic (PLEG): container finished" podID="78a3d980-99f0-4771-a933-056cd7a9225c" containerID="e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536" exitCode=0 Nov 01 15:57:07.128208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.128042 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" Nov 01 15:57:07.128208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.128051 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" event={"ID":"78a3d980-99f0-4771-a933-056cd7a9225c","Type":"ContainerDied","Data":"e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536"} Nov 01 15:57:07.128208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.128105 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5f8877f69-h4kjb" event={"ID":"78a3d980-99f0-4771-a933-056cd7a9225c","Type":"ContainerDied","Data":"41a84cc8686e2cae9a510addc205ef36f58b068872c3d9585bfa553328b60860"} Nov 01 15:57:07.128208 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.128126 5473 scope.go:122] "RemoveContainer" containerID="e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536" Nov 01 15:57:07.137195 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.137178 5473 scope.go:122] "RemoveContainer" containerID="e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536" Nov 01 15:57:07.137503 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:57:07.137478 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536\": container with ID starting with e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536 not found: ID does not exist" containerID="e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536" Nov 01 15:57:07.137604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.137512 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536"} err="failed to get container status \"e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536\": rpc error: code = NotFound desc = could not find container \"e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536\": container with ID starting with e35e3eaad07bcfcf9077cdca72a3bc54c068b804d78493b7cb25f2a0b779f536 not found: ID does not exist" Nov 01 15:57:07.293254 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.293204 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5f8877f69-h4kjb"] Nov 01 15:57:07.300965 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:07.300940 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-5f8877f69-h4kjb"] Nov 01 15:57:09.084057 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:09.084014 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a3d980-99f0-4771-a933-056cd7a9225c" path="/var/lib/kubelet/pods/78a3d980-99f0-4771-a933-056cd7a9225c/volumes" Nov 01 15:57:14.871186 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:14.871152 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:31.106593 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:57:31.106557 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:57:42.048228 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:42.048181 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl"] Nov 01 15:57:42.097667 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:42.097640 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-5hbpl"] Nov 01 15:57:43.080836 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:43.080798 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca42330-9af9-4ba9-80ad-5cc47880102c" path="/var/lib/kubelet/pods/fca42330-9af9-4ba9-80ad-5cc47880102c/volumes" Nov 01 15:57:44.844009 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:44.843977 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:57:44.874060 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:44.874036 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 15:57:52.177034 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:57:52.176995 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:01.107325 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:58:01.107289 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:58:02.047339 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:02.047303 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c"] Nov 01 15:58:02.096993 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:02.096959 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-twg7c"] Nov 01 15:58:03.081434 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:03.081398 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa15ae82-a2a8-4138-8cdb-ebdf5467f101" path="/var/lib/kubelet/pods/fa15ae82-a2a8-4138-8cdb-ebdf5467f101/volumes" Nov 01 15:58:03.104871 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:03.104837 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:15.194820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:15.194772 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:27.048788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:27.048701 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9"] Nov 01 15:58:27.098490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:27.098455 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81566d09-46vd9"] Nov 01 15:58:29.080360 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:29.080326 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b73c6a-5cef-4d8a-a3b5-1b8636774e1e" path="/var/lib/kubelet/pods/52b73c6a-5cef-4d8a-a3b5-1b8636774e1e/volumes" Nov 01 15:58:31.107718 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:58:31.107680 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:58:37.820363 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.820333 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd"] Nov 01 15:58:37.820841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.820709 5473 cpu_manager.go:414] "containerMap: removing container" podUID="78a3d980-99f0-4771-a933-056cd7a9225c" containerName="payload-pre-processing" Nov 01 15:58:37.820841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.820719 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a3d980-99f0-4771-a933-056cd7a9225c" containerName="payload-pre-processing" Nov 01 15:58:37.820841 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.820820 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a3d980-99f0-4771-a933-056cd7a9225c" containerName="payload-pre-processing" Nov 01 15:58:37.824142 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.824125 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.848415 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.848390 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-877f4\" (UniqueName: \"kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.848531 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.848435 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.848608 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.848588 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.880138 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.880092 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd"] Nov 01 15:58:37.927547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.927518 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:58:37.927547 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.927542 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:58:37.949046 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.949025 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.949161 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.949062 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-877f4\" (UniqueName: \"kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.949267 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.949250 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.949820 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.949797 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:37.951345 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:37.951326 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:38.005766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:38.005736 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-877f4\" (UniqueName: \"kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4\") pod \"maas-api-revoke-sub-e2547622-jcvbd\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:38.136975 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:38.136895 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:38.506105 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:38.506063 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd"] Nov 01 15:58:38.507771 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:58:38.507731 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf80574a8_8ae6_46de_8964_804ef1ff69bc.slice/crio-e597b8b965a9e64154f312ec73041e70e38a20a834038ab1c790a42a233e7a1d WatchSource:0}: Error finding container e597b8b965a9e64154f312ec73041e70e38a20a834038ab1c790a42a233e7a1d: Status 404 returned error can't find the container with id e597b8b965a9e64154f312ec73041e70e38a20a834038ab1c790a42a233e7a1d Nov 01 15:58:38.683514 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:38.683481 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" event={"ID":"f80574a8-8ae6-46de-8964-804ef1ff69bc","Type":"ContainerStarted","Data":"e597b8b965a9e64154f312ec73041e70e38a20a834038ab1c790a42a233e7a1d"} Nov 01 15:58:39.689961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:39.689866 5473 generic.go:356] "Generic (PLEG): container finished" podID="f80574a8-8ae6-46de-8964-804ef1ff69bc" containerID="65f9036930ff8024afb0c9fa1836b7aaac96d2e0e9c8727367533e0c87bb40fe" exitCode=0 Nov 01 15:58:39.690423 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:39.689953 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" event={"ID":"f80574a8-8ae6-46de-8964-804ef1ff69bc","Type":"ContainerDied","Data":"65f9036930ff8024afb0c9fa1836b7aaac96d2e0e9c8727367533e0c87bb40fe"} Nov 01 15:58:40.832300 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.832275 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:40.879428 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.879402 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4\" (UniqueName: \"kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4\") pod \"f80574a8-8ae6-46de-8964-804ef1ff69bc\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " Nov 01 15:58:40.879570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.879461 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca\") pod \"f80574a8-8ae6-46de-8964-804ef1ff69bc\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " Nov 01 15:58:40.879570 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.879495 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca\") pod \"f80574a8-8ae6-46de-8964-804ef1ff69bc\" (UID: \"f80574a8-8ae6-46de-8964-804ef1ff69bc\") " Nov 01 15:58:40.879828 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.879801 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca" pod "f80574a8-8ae6-46de-8964-804ef1ff69bc" (UID: "f80574a8-8ae6-46de-8964-804ef1ff69bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:58:40.881522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.881491 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4" pod "f80574a8-8ae6-46de-8964-804ef1ff69bc" (UID: "f80574a8-8ae6-46de-8964-804ef1ff69bc"). InnerVolumeSpecName "kube-api-access-877f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:58:40.881522 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.881509 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca" pod "f80574a8-8ae6-46de-8964-804ef1ff69bc" (UID: "f80574a8-8ae6-46de-8964-804ef1ff69bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:58:40.980926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.980856 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-877f4\" (UniqueName: \"kubernetes.io/projected/f80574a8-8ae6-46de-8964-804ef1ff69bc-kube-api-access-877f4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:58:40.980926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.980881 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:58:40.980926 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:40.980893 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f80574a8-8ae6-46de-8964-804ef1ff69bc-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:58:41.701201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:41.701168 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 15:58:41.701201 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:41.701189 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" event={"ID":"f80574a8-8ae6-46de-8964-804ef1ff69bc","Type":"ContainerDied","Data":"e597b8b965a9e64154f312ec73041e70e38a20a834038ab1c790a42a233e7a1d"} Nov 01 15:58:41.701448 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:41.701264 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e597b8b965a9e64154f312ec73041e70e38a20a834038ab1c790a42a233e7a1d" Nov 01 15:58:45.563604 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:45.563564 5473 scope.go:122] "RemoveContainer" containerID="cc2c8e340fbaaf22ece06b775d23c5c9ec7d7452424c5bcdec8153aac5c9b44f" Nov 01 15:58:49.474392 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:49.474363 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:49.639067 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:49.639034 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:49.959920 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:58:49.959887 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:01.106335 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:59:01.106300 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:59:07.318367 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.318325 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2"] Nov 01 15:59:07.318957 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.318935 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f80574a8-8ae6-46de-8964-804ef1ff69bc" containerName="revoke-keys" Nov 01 15:59:07.319010 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.318962 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f80574a8-8ae6-46de-8964-804ef1ff69bc" containerName="revoke-keys" Nov 01 15:59:07.319110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.319097 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f80574a8-8ae6-46de-8964-804ef1ff69bc" containerName="revoke-keys" Nov 01 15:59:07.322718 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.322700 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.380890 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.380860 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.381058 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.380917 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bf9\" (UniqueName: \"kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.381110 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.381048 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.425425 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.425388 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:59:07.425711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.425689 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:59:07.427827 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.427806 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2"] Nov 01 15:59:07.482429 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.482391 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bf9\" (UniqueName: \"kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.482599 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.482565 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.482666 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.482620 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.483172 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.483154 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.485200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.485174 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.539581 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.539548 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bf9\" (UniqueName: \"kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9\") pod \"maas-api-revoke-sub-c1e271a9-vpxw2\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:07.635313 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:07.635246 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:08.005408 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:08.005383 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2"] Nov 01 15:59:08.006997 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:59:08.006962 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e318ec_c81a_40af_a9ee_327e1573e272.slice/crio-9a8efb66e09d452796a01534977cb88586e5773e8336eed6738f650b3b5510aa WatchSource:0}: Error finding container 9a8efb66e09d452796a01534977cb88586e5773e8336eed6738f650b3b5510aa: Status 404 returned error can't find the container with id 9a8efb66e09d452796a01534977cb88586e5773e8336eed6738f650b3b5510aa Nov 01 15:59:08.908341 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:08.908306 5473 generic.go:356] "Generic (PLEG): container finished" podID="27e318ec-c81a-40af-a9ee-327e1573e272" containerID="e3fe04c63c4c374c20512b2561493ab7c730eb5e3bfe960405cdc68e539f7aaf" exitCode=0 Nov 01 15:59:08.908759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:08.908386 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" event={"ID":"27e318ec-c81a-40af-a9ee-327e1573e272","Type":"ContainerDied","Data":"e3fe04c63c4c374c20512b2561493ab7c730eb5e3bfe960405cdc68e539f7aaf"} Nov 01 15:59:08.908759 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:08.908433 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" event={"ID":"27e318ec-c81a-40af-a9ee-327e1573e272","Type":"ContainerStarted","Data":"9a8efb66e09d452796a01534977cb88586e5773e8336eed6738f650b3b5510aa"} Nov 01 15:59:10.049668 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.049643 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:10.110466 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.110439 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca\") pod \"27e318ec-c81a-40af-a9ee-327e1573e272\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " Nov 01 15:59:10.110587 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.110497 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9\" (UniqueName: \"kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9\") pod \"27e318ec-c81a-40af-a9ee-327e1573e272\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " Nov 01 15:59:10.110691 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.110677 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca\") pod \"27e318ec-c81a-40af-a9ee-327e1573e272\" (UID: \"27e318ec-c81a-40af-a9ee-327e1573e272\") " Nov 01 15:59:10.110783 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.110759 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca" pod "27e318ec-c81a-40af-a9ee-327e1573e272" (UID: "27e318ec-c81a-40af-a9ee-327e1573e272"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:59:10.110913 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.110900 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:10.112597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.112574 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca" pod "27e318ec-c81a-40af-a9ee-327e1573e272" (UID: "27e318ec-c81a-40af-a9ee-327e1573e272"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:59:10.112711 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.112698 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9" pod "27e318ec-c81a-40af-a9ee-327e1573e272" (UID: "27e318ec-c81a-40af-a9ee-327e1573e272"). InnerVolumeSpecName "kube-api-access-j5bf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:59:10.212070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.211997 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27e318ec-c81a-40af-a9ee-327e1573e272-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:10.212070 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.212029 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5bf9\" (UniqueName: \"kubernetes.io/projected/27e318ec-c81a-40af-a9ee-327e1573e272-kube-api-access-j5bf9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:10.920740 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.920707 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" event={"ID":"27e318ec-c81a-40af-a9ee-327e1573e272","Type":"ContainerDied","Data":"9a8efb66e09d452796a01534977cb88586e5773e8336eed6738f650b3b5510aa"} Nov 01 15:59:10.920915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.920754 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a8efb66e09d452796a01534977cb88586e5773e8336eed6738f650b3b5510aa" Nov 01 15:59:10.920915 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:10.920718 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 15:59:12.836526 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:12.836493 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:15.501200 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:15.501154 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:31.107644 ip-10-0-131-109 kubenswrapper[5473]: E1101 15:59:31.107608 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 15:59:33.025185 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.025149 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267"] Nov 01 15:59:33.025623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.025563 5473 cpu_manager.go:414] "containerMap: removing container" podUID="27e318ec-c81a-40af-a9ee-327e1573e272" containerName="revoke-keys" Nov 01 15:59:33.025623 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.025575 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e318ec-c81a-40af-a9ee-327e1573e272" containerName="revoke-keys" Nov 01 15:59:33.025715 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.025679 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e318ec-c81a-40af-a9ee-327e1573e272" containerName="revoke-keys" Nov 01 15:59:33.028766 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.028746 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.131904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.131872 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:59:33.131904 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.131892 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:59:33.134308 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.134286 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267"] Nov 01 15:59:33.173961 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.173927 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.174120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.173974 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7mn\" (UniqueName: \"kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.174120 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.173994 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.275127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.275087 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7mn\" (UniqueName: \"kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.275127 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.275126 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.275464 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.275291 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.275976 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.275955 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.277650 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.277630 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.332384 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.332349 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7mn\" (UniqueName: \"kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn\") pod \"maas-api-revoke-sub-2e1f5b5f-5w267\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.342122 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.342098 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:33.709396 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:33.709371 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267"] Nov 01 15:59:33.711776 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:59:33.711746 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2576f13e_473f_473c_9496_1e0ab018edd3.slice/crio-f8b1c3fdd9d2b9af4ec1b95a95f9638c7f6cb5da075816ee842b7b5697cbc4ea WatchSource:0}: Error finding container f8b1c3fdd9d2b9af4ec1b95a95f9638c7f6cb5da075816ee842b7b5697cbc4ea: Status 404 returned error can't find the container with id f8b1c3fdd9d2b9af4ec1b95a95f9638c7f6cb5da075816ee842b7b5697cbc4ea Nov 01 15:59:34.056002 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:34.055974 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" event={"ID":"2576f13e-473f-473c-9496-1e0ab018edd3","Type":"ContainerStarted","Data":"4d7c3cf8d792e0a5a19a224cf079f3d0767501e96e183f82426b75154c7e717f"} Nov 01 15:59:34.056376 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:34.056034 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" event={"ID":"2576f13e-473f-473c-9496-1e0ab018edd3","Type":"ContainerStarted","Data":"f8b1c3fdd9d2b9af4ec1b95a95f9638c7f6cb5da075816ee842b7b5697cbc4ea"} Nov 01 15:59:34.170191 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:34.170107 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" podStartSLOduration=2.170090265 podStartE2EDuration="2.170090265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:34.167747009 +0000 UTC m=+6113.225898839" watchObservedRunningTime="2026-11-01 15:59:34.170090265 +0000 UTC m=+6113.228242118" Nov 01 15:59:35.063598 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:35.063545 5473 generic.go:356] "Generic (PLEG): container finished" podID="2576f13e-473f-473c-9496-1e0ab018edd3" containerID="4d7c3cf8d792e0a5a19a224cf079f3d0767501e96e183f82426b75154c7e717f" exitCode=0 Nov 01 15:59:35.064091 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:35.063623 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" event={"ID":"2576f13e-473f-473c-9496-1e0ab018edd3","Type":"ContainerDied","Data":"4d7c3cf8d792e0a5a19a224cf079f3d0767501e96e183f82426b75154c7e717f"} Nov 01 15:59:36.210077 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.210050 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:36.300636 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.300601 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn\" (UniqueName: \"kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn\") pod \"2576f13e-473f-473c-9496-1e0ab018edd3\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " Nov 01 15:59:36.300788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.300705 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca\") pod \"2576f13e-473f-473c-9496-1e0ab018edd3\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " Nov 01 15:59:36.300788 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.300765 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca\") pod \"2576f13e-473f-473c-9496-1e0ab018edd3\" (UID: \"2576f13e-473f-473c-9496-1e0ab018edd3\") " Nov 01 15:59:36.301075 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.301051 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca" pod "2576f13e-473f-473c-9496-1e0ab018edd3" (UID: "2576f13e-473f-473c-9496-1e0ab018edd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:59:36.302737 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.302715 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca" pod "2576f13e-473f-473c-9496-1e0ab018edd3" (UID: "2576f13e-473f-473c-9496-1e0ab018edd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:59:36.302822 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.302719 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn" pod "2576f13e-473f-473c-9496-1e0ab018edd3" (UID: "2576f13e-473f-473c-9496-1e0ab018edd3"). InnerVolumeSpecName "kube-api-access-9b7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:59:36.402590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.402514 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:36.402590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.402550 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2576f13e-473f-473c-9496-1e0ab018edd3-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:36.402590 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:36.402566 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7mn\" (UniqueName: \"kubernetes.io/projected/2576f13e-473f-473c-9496-1e0ab018edd3-kube-api-access-9b7mn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:37.075405 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:37.075373 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" Nov 01 15:59:37.080938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:37.080908 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267" event={"ID":"2576f13e-473f-473c-9496-1e0ab018edd3","Type":"ContainerDied","Data":"f8b1c3fdd9d2b9af4ec1b95a95f9638c7f6cb5da075816ee842b7b5697cbc4ea"} Nov 01 15:59:37.080938 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:37.080950 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b1c3fdd9d2b9af4ec1b95a95f9638c7f6cb5da075816ee842b7b5697cbc4ea" Nov 01 15:59:45.227639 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.227607 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz"] Nov 01 15:59:45.230159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.227995 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2576f13e-473f-473c-9496-1e0ab018edd3" containerName="revoke-keys" Nov 01 15:59:45.230159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.228014 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2576f13e-473f-473c-9496-1e0ab018edd3" containerName="revoke-keys" Nov 01 15:59:45.230159 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.228100 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2576f13e-473f-473c-9496-1e0ab018edd3" containerName="revoke-keys" Nov 01 15:59:45.231134 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.231115 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.282615 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.282576 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.282792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.282648 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.282792 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.282753 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b29zn\" (UniqueName: \"kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.287803 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.287775 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz"] Nov 01 15:59:45.334635 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.334610 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 15:59:45.334762 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.334686 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 15:59:45.383797 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.383771 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.383969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.383857 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.383969 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.383927 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29zn\" (UniqueName: \"kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.384539 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.384517 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.386048 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.386031 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.440593 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.440563 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29zn\" (UniqueName: \"kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn\") pod \"maas-api-revoke-sub-ec843018-q6kjz\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.546274 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.546236 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:45.586474 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.586447 5473 scope.go:122] "RemoveContainer" containerID="b32c67f11b4cc8d1c65e4e4a790834119b29e06a42d589b618939011c0a486df" Nov 01 15:59:45.596445 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.596417 5473 scope.go:122] "RemoveContainer" containerID="6aa6ab20f9bd938aa2486a4be8189bc86aa49460fed82b4c542a042723498d92" Nov 01 15:59:45.918041 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:45.918013 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz"] Nov 01 15:59:45.919004 ip-10-0-131-109 kubenswrapper[5473]: W1101 15:59:45.918975 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc239fcb6_18d5_49aa_ba69_92a3449761b8.slice/crio-276e8b01cea24fb9f5915df958ca35d19229b0936af08ff2b5e961294d9ce142 WatchSource:0}: Error finding container 276e8b01cea24fb9f5915df958ca35d19229b0936af08ff2b5e961294d9ce142: Status 404 returned error can't find the container with id 276e8b01cea24fb9f5915df958ca35d19229b0936af08ff2b5e961294d9ce142 Nov 01 15:59:46.122934 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:46.122900 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" event={"ID":"c239fcb6-18d5-49aa-ba69-92a3449761b8","Type":"ContainerStarted","Data":"276e8b01cea24fb9f5915df958ca35d19229b0936af08ff2b5e961294d9ce142"} Nov 01 15:59:47.129628 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:47.129590 5473 generic.go:356] "Generic (PLEG): container finished" podID="c239fcb6-18d5-49aa-ba69-92a3449761b8" containerID="6fca6d4f0a33444ef6c56f16f8912bc00c7794e7ca2c5940c7f5e5a1c7b826e4" exitCode=0 Nov 01 15:59:47.130023 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:47.129674 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" event={"ID":"c239fcb6-18d5-49aa-ba69-92a3449761b8","Type":"ContainerDied","Data":"6fca6d4f0a33444ef6c56f16f8912bc00c7794e7ca2c5940c7f5e5a1c7b826e4"} Nov 01 15:59:48.270381 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.270359 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:48.313490 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.313469 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn\" (UniqueName: \"kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn\") pod \"c239fcb6-18d5-49aa-ba69-92a3449761b8\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " Nov 01 15:59:48.313595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.313520 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca\") pod \"c239fcb6-18d5-49aa-ba69-92a3449761b8\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " Nov 01 15:59:48.313595 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.313544 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca\") pod \"c239fcb6-18d5-49aa-ba69-92a3449761b8\" (UID: \"c239fcb6-18d5-49aa-ba69-92a3449761b8\") " Nov 01 15:59:48.313947 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.313922 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca" pod "c239fcb6-18d5-49aa-ba69-92a3449761b8" (UID: "c239fcb6-18d5-49aa-ba69-92a3449761b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:59:48.315501 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.315474 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn" pod "c239fcb6-18d5-49aa-ba69-92a3449761b8" (UID: "c239fcb6-18d5-49aa-ba69-92a3449761b8"). InnerVolumeSpecName "kube-api-access-b29zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:59:48.315597 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.315522 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca" pod "c239fcb6-18d5-49aa-ba69-92a3449761b8" (UID: "c239fcb6-18d5-49aa-ba69-92a3449761b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:59:48.414921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.414850 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b29zn\" (UniqueName: \"kubernetes.io/projected/c239fcb6-18d5-49aa-ba69-92a3449761b8-kube-api-access-b29zn\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:48.414921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.414876 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:48.414921 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:48.414888 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c239fcb6-18d5-49aa-ba69-92a3449761b8-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 15:59:49.142931 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:49.142905 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" Nov 01 15:59:49.143108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:49.142904 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz" event={"ID":"c239fcb6-18d5-49aa-ba69-92a3449761b8","Type":"ContainerDied","Data":"276e8b01cea24fb9f5915df958ca35d19229b0936af08ff2b5e961294d9ce142"} Nov 01 15:59:49.143108 ip-10-0-131-109 kubenswrapper[5473]: I1101 15:59:49.143030 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="276e8b01cea24fb9f5915df958ca35d19229b0936af08ff2b5e961294d9ce142" Nov 01 16:00:00.154247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.154118 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql"] Nov 01 16:00:00.154782 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.154534 5473 cpu_manager.go:414] "containerMap: removing container" podUID="c239fcb6-18d5-49aa-ba69-92a3449761b8" containerName="revoke-keys" Nov 01 16:00:00.154782 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.154547 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c239fcb6-18d5-49aa-ba69-92a3449761b8" containerName="revoke-keys" Nov 01 16:00:00.154782 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.154616 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="c239fcb6-18d5-49aa-ba69-92a3449761b8" containerName="revoke-keys" Nov 01 16:00:00.158626 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.158593 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.214191 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.214152 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql"] Nov 01 16:00:00.226823 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.226787 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxdp\" (UniqueName: \"kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp\") pod \"maas-api-key-cleanup-29892480-vtvql\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.226984 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.226963 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892480-vtvql\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.260982 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.260955 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:00:00.328190 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.328159 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxdp\" (UniqueName: \"kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp\") pod \"maas-api-key-cleanup-29892480-vtvql\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.328371 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.328264 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892480-vtvql\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.328862 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.328843 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892480-vtvql\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.384846 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.384818 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxdp\" (UniqueName: \"kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp\") pod \"maas-api-key-cleanup-29892480-vtvql\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.473157 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.473077 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:00.844314 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:00.844276 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql"] Nov 01 16:00:00.845392 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:00:00.845361 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b1c68d_cb91_4bab_9622_91ef34492be1.slice/crio-6e30037c741d6883073ba5ad88c367c551550b5c3d9a2041b20e2dc16a107ae2 WatchSource:0}: Error finding container 6e30037c741d6883073ba5ad88c367c551550b5c3d9a2041b20e2dc16a107ae2: Status 404 returned error can't find the container with id 6e30037c741d6883073ba5ad88c367c551550b5c3d9a2041b20e2dc16a107ae2 Nov 01 16:00:01.107449 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:00:01.107362 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:00:01.206093 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:01.206057 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerStarted","Data":"c6c4b36df6c30f562d85d20ae41527491daf8ad4a7690cc5cb1dc68be4122a74"} Nov 01 16:00:01.206558 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:01.206120 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerStarted","Data":"6e30037c741d6883073ba5ad88c367c551550b5c3d9a2041b20e2dc16a107ae2"} Nov 01 16:00:01.318474 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:01.318398 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" podStartSLOduration=1.318368448 podStartE2EDuration="1.318368448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:01.31765995 +0000 UTC m=+6140.375811801" watchObservedRunningTime="2026-11-01 16:00:01.318368448 +0000 UTC m=+6140.376520297" Nov 01 16:00:02.212836 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:02.212804 5473 generic.go:356] "Generic (PLEG): container finished" podID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerID="c6c4b36df6c30f562d85d20ae41527491daf8ad4a7690cc5cb1dc68be4122a74" exitCode=60 Nov 01 16:00:02.213308 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:02.212874 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerDied","Data":"c6c4b36df6c30f562d85d20ae41527491daf8ad4a7690cc5cb1dc68be4122a74"} Nov 01 16:00:02.213308 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:02.213103 5473 scope.go:122] "RemoveContainer" containerID="c6c4b36df6c30f562d85d20ae41527491daf8ad4a7690cc5cb1dc68be4122a74" Nov 01 16:00:03.223909 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:03.223864 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerStarted","Data":"0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba"} Nov 01 16:00:04.274247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:04.273011 5473 generic.go:356] "Generic (PLEG): container finished" podID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerID="0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba" exitCode=60 Nov 01 16:00:04.274247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:04.273206 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerDied","Data":"0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba"} Nov 01 16:00:04.274247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:04.273296 5473 scope.go:122] "RemoveContainer" containerID="c6c4b36df6c30f562d85d20ae41527491daf8ad4a7690cc5cb1dc68be4122a74" Nov 01 16:00:04.274247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:04.273892 5473 scope.go:122] "RemoveContainer" containerID="0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba" Nov 01 16:00:04.282247 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:00:04.277434 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892480-vtvql_odh-ai-gateway-infra(33b1c68d-cb91-4bab-9622-91ef34492be1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" Nov 01 16:00:05.282426 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:05.282394 5473 scope.go:122] "RemoveContainer" containerID="0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba" Nov 01 16:00:05.283455 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:00:05.283421 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892480-vtvql_odh-ai-gateway-infra(33b1c68d-cb91-4bab-9622-91ef34492be1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" Nov 01 16:00:14.476484 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:14.476448 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:15.075250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:15.075209 5473 scope.go:122] "RemoveContainer" containerID="0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba" Nov 01 16:00:15.823931 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:15.823899 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:16.348045 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:16.348014 5473 generic.go:356] "Generic (PLEG): container finished" podID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerID="34b5fe5d0d28f65437d691e1988a79e8d13e6d72f4c71c4b266175122e74e6f7" exitCode=60 Nov 01 16:00:16.348268 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:16.348078 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerDied","Data":"34b5fe5d0d28f65437d691e1988a79e8d13e6d72f4c71c4b266175122e74e6f7"} Nov 01 16:00:16.348268 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:16.348134 5473 scope.go:122] "RemoveContainer" containerID="0c027c4c1e6504c203974bc8c92cf53ec6fac4f7868c5cdcf71f90161ea61bba" Nov 01 16:00:16.348523 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:16.348505 5473 scope.go:122] "RemoveContainer" containerID="34b5fe5d0d28f65437d691e1988a79e8d13e6d72f4c71c4b266175122e74e6f7" Nov 01 16:00:16.348794 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:00:16.348772 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892480-vtvql_odh-ai-gateway-infra(33b1c68d-cb91-4bab-9622-91ef34492be1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" Nov 01 16:00:17.472329 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.472277 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql"] Nov 01 16:00:17.608208 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.608184 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:17.682388 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.682356 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca\") pod \"33b1c68d-cb91-4bab-9622-91ef34492be1\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " Nov 01 16:00:17.682521 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.682478 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp\" (UniqueName: \"kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp\") pod \"33b1c68d-cb91-4bab-9622-91ef34492be1\" (UID: \"33b1c68d-cb91-4bab-9622-91ef34492be1\") " Nov 01 16:00:17.682700 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.682679 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca" pod "33b1c68d-cb91-4bab-9622-91ef34492be1" (UID: "33b1c68d-cb91-4bab-9622-91ef34492be1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:17.682783 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.682769 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33b1c68d-cb91-4bab-9622-91ef34492be1-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:17.684624 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.684598 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp" pod "33b1c68d-cb91-4bab-9622-91ef34492be1" (UID: "33b1c68d-cb91-4bab-9622-91ef34492be1"). InnerVolumeSpecName "kube-api-access-2vxdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:17.783539 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:17.783506 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxdp\" (UniqueName: \"kubernetes.io/projected/33b1c68d-cb91-4bab-9622-91ef34492be1-kube-api-access-2vxdp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:18.361146 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:18.361115 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" Nov 01 16:00:18.361362 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:18.361147 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql" event={"ID":"33b1c68d-cb91-4bab-9622-91ef34492be1","Type":"ContainerDied","Data":"6e30037c741d6883073ba5ad88c367c551550b5c3d9a2041b20e2dc16a107ae2"} Nov 01 16:00:18.361362 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:18.361201 5473 scope.go:122] "RemoveContainer" containerID="34b5fe5d0d28f65437d691e1988a79e8d13e6d72f4c71c4b266175122e74e6f7" Nov 01 16:00:18.526862 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:18.526826 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql"] Nov 01 16:00:18.576687 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:18.576655 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-vtvql"] Nov 01 16:00:19.080466 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.080427 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" path="/var/lib/kubelet/pods/33b1c68d-cb91-4bab-9622-91ef34492be1/volumes" Nov 01 16:00:19.773853 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.773814 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd"] Nov 01 16:00:19.774441 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.774421 5473 cpu_manager.go:414] "containerMap: removing container" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:19.774535 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.774443 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:19.774535 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.774478 5473 cpu_manager.go:414] "containerMap: removing container" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:19.774535 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.774486 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:19.774704 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.774615 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:19.774704 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.774629 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:19.779150 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.779127 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.805109 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.805083 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.805248 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.805226 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.805322 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.805256 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnrcs\" (UniqueName: \"kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.832456 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.832428 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:19.835512 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.835485 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd"] Nov 01 16:00:19.881974 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.881947 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:00:19.905945 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.905914 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.906099 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.906009 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.906099 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.906032 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnrcs\" (UniqueName: \"kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.906762 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.906743 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.908272 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.908254 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:19.963522 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:19.963494 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnrcs\" (UniqueName: \"kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs\") pod \"maas-api-revoke-sub-046e1343-v25qd\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:20.091593 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:20.091522 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:20.462719 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:20.462689 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd"] Nov 01 16:00:20.463884 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:00:20.463851 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dce9d9a_c09e_4f4b_9494_0899dddb6c47.slice/crio-e7881c3f9ab078957d4608a67c76855657801ac618ec6f722d4562143aa6f8dc WatchSource:0}: Error finding container e7881c3f9ab078957d4608a67c76855657801ac618ec6f722d4562143aa6f8dc: Status 404 returned error can't find the container with id e7881c3f9ab078957d4608a67c76855657801ac618ec6f722d4562143aa6f8dc Nov 01 16:00:21.380542 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:21.380503 5473 generic.go:356] "Generic (PLEG): container finished" podID="0dce9d9a-c09e-4f4b-9494-0899dddb6c47" containerID="6f3bba3f93dc6d5c5da752561aa9e9f21f8a7c3f596fd7c5f3d0a78bc344cd26" exitCode=0 Nov 01 16:00:21.380932 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:21.380547 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" event={"ID":"0dce9d9a-c09e-4f4b-9494-0899dddb6c47","Type":"ContainerDied","Data":"6f3bba3f93dc6d5c5da752561aa9e9f21f8a7c3f596fd7c5f3d0a78bc344cd26"} Nov 01 16:00:21.380932 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:21.380600 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" event={"ID":"0dce9d9a-c09e-4f4b-9494-0899dddb6c47","Type":"ContainerStarted","Data":"e7881c3f9ab078957d4608a67c76855657801ac618ec6f722d4562143aa6f8dc"} Nov 01 16:00:22.521655 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.521630 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:22.632932 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.632904 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs\" (UniqueName: \"kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs\") pod \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " Nov 01 16:00:22.633075 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.632939 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca\") pod \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " Nov 01 16:00:22.633075 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.632999 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca\") pod \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\" (UID: \"0dce9d9a-c09e-4f4b-9494-0899dddb6c47\") " Nov 01 16:00:22.633363 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.633337 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca" pod "0dce9d9a-c09e-4f4b-9494-0899dddb6c47" (UID: "0dce9d9a-c09e-4f4b-9494-0899dddb6c47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:22.635353 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.635321 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca" pod "0dce9d9a-c09e-4f4b-9494-0899dddb6c47" (UID: "0dce9d9a-c09e-4f4b-9494-0899dddb6c47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:22.635522 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.635497 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs" pod "0dce9d9a-c09e-4f4b-9494-0899dddb6c47" (UID: "0dce9d9a-c09e-4f4b-9494-0899dddb6c47"). InnerVolumeSpecName "kube-api-access-tnrcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:22.733765 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.733734 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:22.733765 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.733759 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnrcs\" (UniqueName: \"kubernetes.io/projected/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-kube-api-access-tnrcs\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:22.733765 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:22.733769 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce9d9a-c09e-4f4b-9494-0899dddb6c47-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.392496 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:23.392411 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" event={"ID":"0dce9d9a-c09e-4f4b-9494-0899dddb6c47","Type":"ContainerDied","Data":"e7881c3f9ab078957d4608a67c76855657801ac618ec6f722d4562143aa6f8dc"} Nov 01 16:00:23.392496 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:23.392462 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7881c3f9ab078957d4608a67c76855657801ac618ec6f722d4562143aa6f8dc" Nov 01 16:00:23.392496 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:23.392424 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd" Nov 01 16:00:30.481740 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.481709 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw"] Nov 01 16:00:30.482140 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.482120 5473 cpu_manager.go:414] "containerMap: removing container" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:30.482140 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.482132 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:30.482140 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.482140 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0dce9d9a-c09e-4f4b-9494-0899dddb6c47" containerName="revoke-keys" Nov 01 16:00:30.482285 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.482147 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dce9d9a-c09e-4f4b-9494-0899dddb6c47" containerName="revoke-keys" Nov 01 16:00:30.482285 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.482242 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dce9d9a-c09e-4f4b-9494-0899dddb6c47" containerName="revoke-keys" Nov 01 16:00:30.482285 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.482253 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b1c68d-cb91-4bab-9622-91ef34492be1" containerName="cleanup" Nov 01 16:00:30.485542 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.485527 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.542298 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.542267 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw"] Nov 01 16:00:30.588966 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.588936 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:30.589113 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.589081 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:00:30.611538 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.611510 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.611648 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.611575 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.611699 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.611681 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc4rp\" (UniqueName: \"kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.712418 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.712387 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.712577 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.712449 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.712577 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.712520 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc4rp\" (UniqueName: \"kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.713134 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.713108 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.714743 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.714721 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.769692 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.769669 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc4rp\" (UniqueName: \"kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp\") pod \"maas-api-revoke-sub-d3fba056-ndhlw\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:30.798503 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:30.798483 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:31.107411 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:00:31.107329 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:00:31.167407 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:31.167381 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw"] Nov 01 16:00:31.168714 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:00:31.168689 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b36878_ec14_4546_be91_f8548400db00.slice/crio-758d6dd61401913c3e77bfa3ece25f51da7e7744d82165abafbf3ffe1181054d WatchSource:0}: Error finding container 758d6dd61401913c3e77bfa3ece25f51da7e7744d82165abafbf3ffe1181054d: Status 404 returned error can't find the container with id 758d6dd61401913c3e77bfa3ece25f51da7e7744d82165abafbf3ffe1181054d Nov 01 16:00:31.437268 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:31.437210 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" event={"ID":"67b36878-ec14-4546-be91-f8548400db00","Type":"ContainerStarted","Data":"758d6dd61401913c3e77bfa3ece25f51da7e7744d82165abafbf3ffe1181054d"} Nov 01 16:00:32.443437 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:32.443404 5473 generic.go:356] "Generic (PLEG): container finished" podID="67b36878-ec14-4546-be91-f8548400db00" containerID="84d94ee22c4ba61a70a9a7b6a1757f1e6395d3590819ff3757f1ae811038164a" exitCode=0 Nov 01 16:00:32.443821 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:32.443487 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" event={"ID":"67b36878-ec14-4546-be91-f8548400db00","Type":"ContainerDied","Data":"84d94ee22c4ba61a70a9a7b6a1757f1e6395d3590819ff3757f1ae811038164a"} Nov 01 16:00:33.727558 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.727485 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:00:33.850596 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.850553 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca\") pod \"67b36878-ec14-4546-be91-f8548400db00\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " Nov 01 16:00:33.850920 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.850896 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp\" (UniqueName: \"kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp\") pod \"67b36878-ec14-4546-be91-f8548400db00\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " Nov 01 16:00:33.851081 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.851062 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca\") pod \"67b36878-ec14-4546-be91-f8548400db00\" (UID: \"67b36878-ec14-4546-be91-f8548400db00\") " Nov 01 16:00:33.855426 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.855398 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca" pod "67b36878-ec14-4546-be91-f8548400db00" (UID: "67b36878-ec14-4546-be91-f8548400db00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:33.874239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.863041 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp" pod "67b36878-ec14-4546-be91-f8548400db00" (UID: "67b36878-ec14-4546-be91-f8548400db00"). InnerVolumeSpecName "kube-api-access-gc4rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:33.874239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.865615 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca" pod "67b36878-ec14-4546-be91-f8548400db00" (UID: "67b36878-ec14-4546-be91-f8548400db00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:33.973889 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.959880 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67b36878-ec14-4546-be91-f8548400db00-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:33.974120 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.974096 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc4rp\" (UniqueName: \"kubernetes.io/projected/67b36878-ec14-4546-be91-f8548400db00-kube-api-access-gc4rp\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:33.978361 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:33.978304 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67b36878-ec14-4546-be91-f8548400db00-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:34.516239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:34.500096 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" event={"ID":"67b36878-ec14-4546-be91-f8548400db00","Type":"ContainerDied","Data":"758d6dd61401913c3e77bfa3ece25f51da7e7744d82165abafbf3ffe1181054d"} Nov 01 16:00:34.516239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:34.500163 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758d6dd61401913c3e77bfa3ece25f51da7e7744d82165abafbf3ffe1181054d" Nov 01 16:00:34.516239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:00:34.500253 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw" Nov 01 16:01:01.106834 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:01:01.106798 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:01:05.001033 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.000999 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8"] Nov 01 16:01:05.001619 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.001599 5473 cpu_manager.go:414] "containerMap: removing container" podUID="67b36878-ec14-4546-be91-f8548400db00" containerName="revoke-keys" Nov 01 16:01:05.001712 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.001623 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b36878-ec14-4546-be91-f8548400db00" containerName="revoke-keys" Nov 01 16:01:05.001789 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.001751 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b36878-ec14-4546-be91-f8548400db00" containerName="revoke-keys" Nov 01 16:01:05.004779 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.004757 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.109581 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.109545 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:01:05.109764 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.109639 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:01:05.112013 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.111986 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8"] Nov 01 16:01:05.148883 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.148849 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.148883 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.148884 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfqr\" (UniqueName: \"kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.149071 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.148916 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.249961 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.249927 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.249961 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.249967 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfqr\" (UniqueName: \"kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.250200 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.250087 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.250755 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.250728 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.252337 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.252284 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.306574 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.306546 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfqr\" (UniqueName: \"kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr\") pod \"maas-api-revoke-sub-ae594979-rjmg8\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.318363 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.318341 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:05.688407 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:05.688361 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8"] Nov 01 16:01:05.690004 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:01:05.689977 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39de830_66db_4557_8873_315787a975e9.slice/crio-b75d4d1c6e57b8ab1c483e305bb922c40e04bd315eb5a49835620119bbf3d4c6 WatchSource:0}: Error finding container b75d4d1c6e57b8ab1c483e305bb922c40e04bd315eb5a49835620119bbf3d4c6: Status 404 returned error can't find the container with id b75d4d1c6e57b8ab1c483e305bb922c40e04bd315eb5a49835620119bbf3d4c6 Nov 01 16:01:06.699197 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:06.699114 5473 generic.go:356] "Generic (PLEG): container finished" podID="f39de830-66db-4557-8873-315787a975e9" containerID="ee79d07f11b54cb3ae15686c517a2f2f410c40bc6d0a09c9adffee2b0b41d17f" exitCode=0 Nov 01 16:01:06.699197 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:06.699156 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" event={"ID":"f39de830-66db-4557-8873-315787a975e9","Type":"ContainerDied","Data":"ee79d07f11b54cb3ae15686c517a2f2f410c40bc6d0a09c9adffee2b0b41d17f"} Nov 01 16:01:06.699197 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:06.699192 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" event={"ID":"f39de830-66db-4557-8873-315787a975e9","Type":"ContainerStarted","Data":"b75d4d1c6e57b8ab1c483e305bb922c40e04bd315eb5a49835620119bbf3d4c6"} Nov 01 16:01:07.836818 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.836794 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:07.979825 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.979746 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca\") pod \"f39de830-66db-4557-8873-315787a975e9\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " Nov 01 16:01:07.979825 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.979826 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca\") pod \"f39de830-66db-4557-8873-315787a975e9\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " Nov 01 16:01:07.980093 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.979866 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr\" (UniqueName: \"kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr\") pod \"f39de830-66db-4557-8873-315787a975e9\" (UID: \"f39de830-66db-4557-8873-315787a975e9\") " Nov 01 16:01:07.980269 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.980239 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca" pod "f39de830-66db-4557-8873-315787a975e9" (UID: "f39de830-66db-4557-8873-315787a975e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:01:07.981970 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.981942 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca" pod "f39de830-66db-4557-8873-315787a975e9" (UID: "f39de830-66db-4557-8873-315787a975e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:01:07.982078 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:07.982011 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr" pod "f39de830-66db-4557-8873-315787a975e9" (UID: "f39de830-66db-4557-8873-315787a975e9"). InnerVolumeSpecName "kube-api-access-tgfqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:01:08.080798 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:08.080760 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39de830-66db-4557-8873-315787a975e9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:08.080798 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:08.080795 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39de830-66db-4557-8873-315787a975e9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:08.080798 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:08.080804 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgfqr\" (UniqueName: \"kubernetes.io/projected/f39de830-66db-4557-8873-315787a975e9-kube-api-access-tgfqr\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:08.710723 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:08.710688 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" Nov 01 16:01:08.710945 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:08.710727 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8" event={"ID":"f39de830-66db-4557-8873-315787a975e9","Type":"ContainerDied","Data":"b75d4d1c6e57b8ab1c483e305bb922c40e04bd315eb5a49835620119bbf3d4c6"} Nov 01 16:01:08.710945 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:08.710775 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b75d4d1c6e57b8ab1c483e305bb922c40e04bd315eb5a49835620119bbf3d4c6" Nov 01 16:01:16.158234 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:16.158164 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:23.424731 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.424530 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54"] Nov 01 16:01:23.432243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.425865 5473 cpu_manager.go:414] "containerMap: removing container" podUID="f39de830-66db-4557-8873-315787a975e9" containerName="revoke-keys" Nov 01 16:01:23.438723 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.438698 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39de830-66db-4557-8873-315787a975e9" containerName="revoke-keys" Nov 01 16:01:23.439085 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.439065 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39de830-66db-4557-8873-315787a975e9" containerName="revoke-keys" Nov 01 16:01:23.459237 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.447475 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.536238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.532146 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54"] Nov 01 16:01:23.552243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.552059 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.552243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.552187 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtq7\" (UniqueName: \"kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.552488 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.552257 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.577238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.565713 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:01:23.577238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.566478 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:01:23.683850 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.683755 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtq7\" (UniqueName: \"kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.684136 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.684113 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.684376 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.684355 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.696338 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.696314 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.742238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.730905 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.794905 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.794868 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtq7\" (UniqueName: \"kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7\") pod \"maas-api-revoke-sub-a4aef7d7-cnw54\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:23.903612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:23.903335 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:24.703886 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:24.703853 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54"] Nov 01 16:01:24.896250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:24.886502 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" event={"ID":"4ed3bc33-2476-44e8-9aba-15648f6f41fe","Type":"ContainerStarted","Data":"d3b67007ac4b8ecb24909ab700b743d3abb69952818c05021739557e7bf86b6c"} Nov 01 16:01:25.895592 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:25.895564 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" event={"ID":"4ed3bc33-2476-44e8-9aba-15648f6f41fe","Type":"ContainerStarted","Data":"5989a79b6abfb5582308126796ae71b41f836927ca65bf9deee055287e394eba"} Nov 01 16:01:26.902960 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:26.902930 5473 generic.go:356] "Generic (PLEG): container finished" podID="4ed3bc33-2476-44e8-9aba-15648f6f41fe" containerID="5989a79b6abfb5582308126796ae71b41f836927ca65bf9deee055287e394eba" exitCode=0 Nov 01 16:01:26.903444 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:26.903018 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" event={"ID":"4ed3bc33-2476-44e8-9aba-15648f6f41fe","Type":"ContainerDied","Data":"5989a79b6abfb5582308126796ae71b41f836927ca65bf9deee055287e394eba"} Nov 01 16:01:27.033186 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.033161 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:27.145749 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.145717 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca\") pod \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " Nov 01 16:01:27.145749 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.145765 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca\") pod \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " Nov 01 16:01:27.146020 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.145812 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7\" (UniqueName: \"kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7\") pod \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\" (UID: \"4ed3bc33-2476-44e8-9aba-15648f6f41fe\") " Nov 01 16:01:27.146501 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.146399 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca" pod "4ed3bc33-2476-44e8-9aba-15648f6f41fe" (UID: "4ed3bc33-2476-44e8-9aba-15648f6f41fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:01:27.148542 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.148512 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7" pod "4ed3bc33-2476-44e8-9aba-15648f6f41fe" (UID: "4ed3bc33-2476-44e8-9aba-15648f6f41fe"). InnerVolumeSpecName "kube-api-access-gvtq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:01:27.148674 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.148519 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca" pod "4ed3bc33-2476-44e8-9aba-15648f6f41fe" (UID: "4ed3bc33-2476-44e8-9aba-15648f6f41fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:01:27.247170 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.247139 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:27.247170 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.247167 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ed3bc33-2476-44e8-9aba-15648f6f41fe-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:27.247391 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.247177 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvtq7\" (UniqueName: \"kubernetes.io/projected/4ed3bc33-2476-44e8-9aba-15648f6f41fe-kube-api-access-gvtq7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:27.909746 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.909719 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" Nov 01 16:01:27.910161 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.909713 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54" event={"ID":"4ed3bc33-2476-44e8-9aba-15648f6f41fe","Type":"ContainerDied","Data":"d3b67007ac4b8ecb24909ab700b743d3abb69952818c05021739557e7bf86b6c"} Nov 01 16:01:27.910161 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:27.909863 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b67007ac4b8ecb24909ab700b743d3abb69952818c05021739557e7bf86b6c" Nov 01 16:01:31.106838 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:01:31.106803 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:01:44.253124 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.253083 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4"] Nov 01 16:01:44.253803 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.253744 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4ed3bc33-2476-44e8-9aba-15648f6f41fe" containerName="revoke-keys" Nov 01 16:01:44.253803 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.253762 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed3bc33-2476-44e8-9aba-15648f6f41fe" containerName="revoke-keys" Nov 01 16:01:44.253963 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.253859 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed3bc33-2476-44e8-9aba-15648f6f41fe" containerName="revoke-keys" Nov 01 16:01:44.258203 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.258172 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.319049 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.319010 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4"] Nov 01 16:01:44.414610 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.414546 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:01:44.415246 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.415227 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:01:44.420340 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.419789 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.420340 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.419848 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6w7\" (UniqueName: \"kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.420340 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.419890 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.522244 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.520735 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.522244 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.520938 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.522244 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.520981 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6w7\" (UniqueName: \"kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.522244 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.522172 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.546721 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.546646 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.583991 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.583930 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6w7\" (UniqueName: \"kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7\") pod \"maas-api-revoke-sub-79805ca9-fk5w4\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:44.878729 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:44.878384 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:45.566660 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:45.566627 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4"] Nov 01 16:01:46.047021 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:46.046978 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" event={"ID":"e5655804-5f93-4dc7-af39-47624a7cf407","Type":"ContainerStarted","Data":"0c7a95cad01e790f8a1ff4e3b8b72e99304ac2e7304b8188f431d34dd11540f0"} Nov 01 16:01:47.060681 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:47.060641 5473 generic.go:356] "Generic (PLEG): container finished" podID="e5655804-5f93-4dc7-af39-47624a7cf407" containerID="f052bb7cc2a3d0bce109286daf26cac78d8c2220ee9adc3e0bac5e007c64a245" exitCode=0 Nov 01 16:01:47.061139 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:47.060798 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" event={"ID":"e5655804-5f93-4dc7-af39-47624a7cf407","Type":"ContainerDied","Data":"f052bb7cc2a3d0bce109286daf26cac78d8c2220ee9adc3e0bac5e007c64a245"} Nov 01 16:01:48.196498 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.196476 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:48.280645 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.280618 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca\") pod \"e5655804-5f93-4dc7-af39-47624a7cf407\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " Nov 01 16:01:48.280792 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.280689 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7\" (UniqueName: \"kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7\") pod \"e5655804-5f93-4dc7-af39-47624a7cf407\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " Nov 01 16:01:48.280792 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.280769 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca\") pod \"e5655804-5f93-4dc7-af39-47624a7cf407\" (UID: \"e5655804-5f93-4dc7-af39-47624a7cf407\") " Nov 01 16:01:48.281133 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.281110 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca" pod "e5655804-5f93-4dc7-af39-47624a7cf407" (UID: "e5655804-5f93-4dc7-af39-47624a7cf407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:01:48.282778 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.282750 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca" pod "e5655804-5f93-4dc7-af39-47624a7cf407" (UID: "e5655804-5f93-4dc7-af39-47624a7cf407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:01:48.282911 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.282754 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7" pod "e5655804-5f93-4dc7-af39-47624a7cf407" (UID: "e5655804-5f93-4dc7-af39-47624a7cf407"). InnerVolumeSpecName "kube-api-access-ln6w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:01:48.382487 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.382408 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:48.382487 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.382436 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6w7\" (UniqueName: \"kubernetes.io/projected/e5655804-5f93-4dc7-af39-47624a7cf407-kube-api-access-ln6w7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:48.382487 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:48.382450 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5655804-5f93-4dc7-af39-47624a7cf407-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:49.072572 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:49.072533 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" event={"ID":"e5655804-5f93-4dc7-af39-47624a7cf407","Type":"ContainerDied","Data":"0c7a95cad01e790f8a1ff4e3b8b72e99304ac2e7304b8188f431d34dd11540f0"} Nov 01 16:01:49.072572 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:49.072571 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4" Nov 01 16:01:49.072791 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:01:49.072593 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c7a95cad01e790f8a1ff4e3b8b72e99304ac2e7304b8188f431d34dd11540f0" Nov 01 16:02:01.304755 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:02:01.304708 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:02:01.521280 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.521243 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h"] Nov 01 16:02:01.522108 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.522081 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e5655804-5f93-4dc7-af39-47624a7cf407" containerName="revoke-keys" Nov 01 16:02:01.522288 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.522270 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5655804-5f93-4dc7-af39-47624a7cf407" containerName="revoke-keys" Nov 01 16:02:01.522586 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.522567 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5655804-5f93-4dc7-af39-47624a7cf407" containerName="revoke-keys" Nov 01 16:02:01.544518 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.544495 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.575503 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.575067 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnq6f\" (UniqueName: \"kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.575503 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.575130 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.575503 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.575257 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.593237 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.578615 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h"] Nov 01 16:02:01.624635 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.624529 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:02:01.624635 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.624604 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:01.709455 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.709422 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.709845 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.709805 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnq6f\" (UniqueName: \"kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.720416 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.711406 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.721025 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.721003 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.747106 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.747051 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.805248 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.802291 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnq6f\" (UniqueName: \"kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f\") pod \"maas-api-revoke-sub-0b7e8edb-n5g9h\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:01.988464 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:01.988373 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:02.593525 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:02.593497 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h"] Nov 01 16:02:02.594717 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:02:02.594690 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2970462c_af06_4313_8a24_bdc79fad1859.slice/crio-f24655d3001c324c9edcc49c9441b2133a6dcad5bf4779f7709487ffc3b8613d WatchSource:0}: Error finding container f24655d3001c324c9edcc49c9441b2133a6dcad5bf4779f7709487ffc3b8613d: Status 404 returned error can't find the container with id f24655d3001c324c9edcc49c9441b2133a6dcad5bf4779f7709487ffc3b8613d Nov 01 16:02:03.253651 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:03.253553 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" event={"ID":"2970462c-af06-4313-8a24-bdc79fad1859","Type":"ContainerStarted","Data":"88ed0d949e84b36c178700f02cb66548f5303949ca9518919175835f95b57f2f"} Nov 01 16:02:03.253651 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:03.253619 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" event={"ID":"2970462c-af06-4313-8a24-bdc79fad1859","Type":"ContainerStarted","Data":"f24655d3001c324c9edcc49c9441b2133a6dcad5bf4779f7709487ffc3b8613d"} Nov 01 16:02:03.366707 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:03.366651 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" podStartSLOduration=2.366637422 podStartE2EDuration="2.366637422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:03.365995455 +0000 UTC m=+6262.424147324" watchObservedRunningTime="2026-11-01 16:02:03.366637422 +0000 UTC m=+6262.424789309" Nov 01 16:02:04.260960 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:04.260922 5473 generic.go:356] "Generic (PLEG): container finished" podID="2970462c-af06-4313-8a24-bdc79fad1859" containerID="88ed0d949e84b36c178700f02cb66548f5303949ca9518919175835f95b57f2f" exitCode=0 Nov 01 16:02:04.261387 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:04.261008 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" event={"ID":"2970462c-af06-4313-8a24-bdc79fad1859","Type":"ContainerDied","Data":"88ed0d949e84b36c178700f02cb66548f5303949ca9518919175835f95b57f2f"} Nov 01 16:02:05.399106 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.399082 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:05.481243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.481197 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca\") pod \"2970462c-af06-4313-8a24-bdc79fad1859\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " Nov 01 16:02:05.481443 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.481263 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f\" (UniqueName: \"kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f\") pod \"2970462c-af06-4313-8a24-bdc79fad1859\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " Nov 01 16:02:05.481443 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.481423 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca\") pod \"2970462c-af06-4313-8a24-bdc79fad1859\" (UID: \"2970462c-af06-4313-8a24-bdc79fad1859\") " Nov 01 16:02:05.481778 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.481755 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca" pod "2970462c-af06-4313-8a24-bdc79fad1859" (UID: "2970462c-af06-4313-8a24-bdc79fad1859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:05.483348 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.483320 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca" pod "2970462c-af06-4313-8a24-bdc79fad1859" (UID: "2970462c-af06-4313-8a24-bdc79fad1859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:05.483348 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.483341 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f" pod "2970462c-af06-4313-8a24-bdc79fad1859" (UID: "2970462c-af06-4313-8a24-bdc79fad1859"). InnerVolumeSpecName "kube-api-access-fnq6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:05.582153 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.582079 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:05.582153 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.582106 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2970462c-af06-4313-8a24-bdc79fad1859-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:05.582153 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:05.582117 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnq6f\" (UniqueName: \"kubernetes.io/projected/2970462c-af06-4313-8a24-bdc79fad1859-kube-api-access-fnq6f\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:06.273772 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:06.273740 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" Nov 01 16:02:06.273967 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:06.273744 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h" event={"ID":"2970462c-af06-4313-8a24-bdc79fad1859","Type":"ContainerDied","Data":"f24655d3001c324c9edcc49c9441b2133a6dcad5bf4779f7709487ffc3b8613d"} Nov 01 16:02:06.273967 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:06.273868 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f24655d3001c324c9edcc49c9441b2133a6dcad5bf4779f7709487ffc3b8613d" Nov 01 16:02:12.847607 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.847567 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz"] Nov 01 16:02:12.848242 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.848199 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2970462c-af06-4313-8a24-bdc79fad1859" containerName="revoke-keys" Nov 01 16:02:12.848242 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.848234 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2970462c-af06-4313-8a24-bdc79fad1859" containerName="revoke-keys" Nov 01 16:02:12.848425 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.848344 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2970462c-af06-4313-8a24-bdc79fad1859" containerName="revoke-keys" Nov 01 16:02:12.851447 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.851425 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:12.908617 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.908582 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz"] Nov 01 16:02:12.953973 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.953944 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:12.954143 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.954043 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:02:12.961627 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.961598 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:12.961777 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.961633 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dpm\" (UniqueName: \"kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:12.961777 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:12.961672 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.062614 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.062584 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.062614 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.062616 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dpm\" (UniqueName: \"kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.062831 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.062647 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.063238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.063200 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.064919 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.064897 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.119265 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.119175 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dpm\" (UniqueName: \"kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm\") pod \"maas-api-revoke-sub-3d4156f1-5d7dz\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.164509 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.164484 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:13.533832 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:13.533802 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz"] Nov 01 16:02:13.535567 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:02:13.535537 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode583bbb1_063b_477c_b37d_2269cc7ad92f.slice/crio-7d0546c8d5a46e6f702d0c30c17d0634034d564343532316d32fff846e04eaa1 WatchSource:0}: Error finding container 7d0546c8d5a46e6f702d0c30c17d0634034d564343532316d32fff846e04eaa1: Status 404 returned error can't find the container with id 7d0546c8d5a46e6f702d0c30c17d0634034d564343532316d32fff846e04eaa1 Nov 01 16:02:14.321746 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:14.321709 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" event={"ID":"e583bbb1-063b-477c-b37d-2269cc7ad92f","Type":"ContainerStarted","Data":"d1837473d1dc73393abdfe079272fb14cac95a2084a205664004c07135b394ac"} Nov 01 16:02:14.322127 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:14.321795 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" event={"ID":"e583bbb1-063b-477c-b37d-2269cc7ad92f","Type":"ContainerStarted","Data":"7d0546c8d5a46e6f702d0c30c17d0634034d564343532316d32fff846e04eaa1"} Nov 01 16:02:14.436225 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:14.436173 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" podStartSLOduration=2.436157765 podStartE2EDuration="2.436157765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:14.434784127 +0000 UTC m=+6273.492935977" watchObservedRunningTime="2026-11-01 16:02:14.436157765 +0000 UTC m=+6273.494309639" Nov 01 16:02:15.329201 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:15.329167 5473 generic.go:356] "Generic (PLEG): container finished" podID="e583bbb1-063b-477c-b37d-2269cc7ad92f" containerID="d1837473d1dc73393abdfe079272fb14cac95a2084a205664004c07135b394ac" exitCode=0 Nov 01 16:02:15.329633 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:15.329241 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" event={"ID":"e583bbb1-063b-477c-b37d-2269cc7ad92f","Type":"ContainerDied","Data":"d1837473d1dc73393abdfe079272fb14cac95a2084a205664004c07135b394ac"} Nov 01 16:02:16.468454 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.468431 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:16.530502 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.530467 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:16.595492 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.595420 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca\") pod \"e583bbb1-063b-477c-b37d-2269cc7ad92f\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " Nov 01 16:02:16.595492 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.595489 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm\" (UniqueName: \"kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm\") pod \"e583bbb1-063b-477c-b37d-2269cc7ad92f\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " Nov 01 16:02:16.595752 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.595636 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca\") pod \"e583bbb1-063b-477c-b37d-2269cc7ad92f\" (UID: \"e583bbb1-063b-477c-b37d-2269cc7ad92f\") " Nov 01 16:02:16.595826 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.595770 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca" pod "e583bbb1-063b-477c-b37d-2269cc7ad92f" (UID: "e583bbb1-063b-477c-b37d-2269cc7ad92f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:16.595887 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.595872 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:16.597644 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.597608 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca" pod "e583bbb1-063b-477c-b37d-2269cc7ad92f" (UID: "e583bbb1-063b-477c-b37d-2269cc7ad92f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:16.597776 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.597702 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm" pod "e583bbb1-063b-477c-b37d-2269cc7ad92f" (UID: "e583bbb1-063b-477c-b37d-2269cc7ad92f"). InnerVolumeSpecName "kube-api-access-t6dpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:16.696934 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.696901 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e583bbb1-063b-477c-b37d-2269cc7ad92f-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:16.696934 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:16.696933 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6dpm\" (UniqueName: \"kubernetes.io/projected/e583bbb1-063b-477c-b37d-2269cc7ad92f-kube-api-access-t6dpm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:17.342115 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:17.342031 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" event={"ID":"e583bbb1-063b-477c-b37d-2269cc7ad92f","Type":"ContainerDied","Data":"7d0546c8d5a46e6f702d0c30c17d0634034d564343532316d32fff846e04eaa1"} Nov 01 16:02:17.342115 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:17.342080 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0546c8d5a46e6f702d0c30c17d0634034d564343532316d32fff846e04eaa1" Nov 01 16:02:17.342115 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:17.342042 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz" Nov 01 16:02:30.707039 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.706997 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz"] Nov 01 16:02:30.709206 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.709181 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5d7dz"] Nov 01 16:02:30.776189 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.776156 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd"] Nov 01 16:02:30.776589 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.776576 5473 cpu_manager.go:414] "containerMap: removing container" podUID="e583bbb1-063b-477c-b37d-2269cc7ad92f" containerName="revoke-keys" Nov 01 16:02:30.776639 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.776590 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e583bbb1-063b-477c-b37d-2269cc7ad92f" containerName="revoke-keys" Nov 01 16:02:30.776677 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.776663 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="e583bbb1-063b-477c-b37d-2269cc7ad92f" containerName="revoke-keys" Nov 01 16:02:30.779969 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.779954 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.792355 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.792331 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972q4\" (UniqueName: \"kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.792503 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.792374 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.792592 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.792570 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.837790 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.837753 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd"] Nov 01 16:02:30.847142 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.847110 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:30.847319 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.847257 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:02:30.893232 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.893190 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972q4\" (UniqueName: \"kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.893386 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.893245 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.893386 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.893321 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.893987 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.893967 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.895612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.895589 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:30.949805 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:30.949776 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-972q4\" (UniqueName: \"kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4\") pod \"maas-api-revoke-sub-3d4156f1-w88pd\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:31.080552 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:31.080519 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e583bbb1-063b-477c-b37d-2269cc7ad92f" path="/var/lib/kubelet/pods/e583bbb1-063b-477c-b37d-2269cc7ad92f/volumes" Nov 01 16:02:31.092807 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:31.092786 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:31.106093 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:02:31.106063 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:02:31.462415 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:31.462386 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd"] Nov 01 16:02:31.463907 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:02:31.463880 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8bba8d_4e50_44c1_bd3e_66a87c8be330.slice/crio-4c8b5dc05c2470025be624564ac6465ff542d96768d5f9b6a0c35ffdc945f16b WatchSource:0}: Error finding container 4c8b5dc05c2470025be624564ac6465ff542d96768d5f9b6a0c35ffdc945f16b: Status 404 returned error can't find the container with id 4c8b5dc05c2470025be624564ac6465ff542d96768d5f9b6a0c35ffdc945f16b Nov 01 16:02:32.453735 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:32.453705 5473 generic.go:356] "Generic (PLEG): container finished" podID="0e8bba8d-4e50-44c1-bd3e-66a87c8be330" containerID="1e5cb9d0f6b2f1272435771686366ada5c7128464a940189659bc590fc402323" exitCode=0 Nov 01 16:02:32.454144 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:32.453783 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" event={"ID":"0e8bba8d-4e50-44c1-bd3e-66a87c8be330","Type":"ContainerDied","Data":"1e5cb9d0f6b2f1272435771686366ada5c7128464a940189659bc590fc402323"} Nov 01 16:02:32.454144 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:32.453851 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" event={"ID":"0e8bba8d-4e50-44c1-bd3e-66a87c8be330","Type":"ContainerStarted","Data":"4c8b5dc05c2470025be624564ac6465ff542d96768d5f9b6a0c35ffdc945f16b"} Nov 01 16:02:33.591123 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.591099 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:33.618378 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.618353 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca\") pod \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " Nov 01 16:02:33.618490 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.618461 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca\") pod \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " Nov 01 16:02:33.618539 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.618522 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4\" (UniqueName: \"kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4\") pod \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\" (UID: \"0e8bba8d-4e50-44c1-bd3e-66a87c8be330\") " Nov 01 16:02:33.618667 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.618650 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca" pod "0e8bba8d-4e50-44c1-bd3e-66a87c8be330" (UID: "0e8bba8d-4e50-44c1-bd3e-66a87c8be330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:33.618857 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.618835 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:33.620527 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.620501 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4" pod "0e8bba8d-4e50-44c1-bd3e-66a87c8be330" (UID: "0e8bba8d-4e50-44c1-bd3e-66a87c8be330"). InnerVolumeSpecName "kube-api-access-972q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:33.620630 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.620572 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca" pod "0e8bba8d-4e50-44c1-bd3e-66a87c8be330" (UID: "0e8bba8d-4e50-44c1-bd3e-66a87c8be330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:33.720092 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.720017 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972q4\" (UniqueName: \"kubernetes.io/projected/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-kube-api-access-972q4\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:33.720092 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:33.720054 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e8bba8d-4e50-44c1-bd3e-66a87c8be330-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:34.467234 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:34.467194 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" event={"ID":"0e8bba8d-4e50-44c1-bd3e-66a87c8be330","Type":"ContainerDied","Data":"4c8b5dc05c2470025be624564ac6465ff542d96768d5f9b6a0c35ffdc945f16b"} Nov 01 16:02:34.467234 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:34.467225 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd" Nov 01 16:02:34.467461 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:34.467252 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8b5dc05c2470025be624564ac6465ff542d96768d5f9b6a0c35ffdc945f16b" Nov 01 16:02:44.886471 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:44.886443 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:02:44.918632 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:02:44.918601 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:03:01.106339 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:03:01.106255 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:03:04.871367 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.871330 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48"] Nov 01 16:03:04.871987 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.871965 5473 cpu_manager.go:414] "containerMap: removing container" podUID="0e8bba8d-4e50-44c1-bd3e-66a87c8be330" containerName="revoke-keys" Nov 01 16:03:04.872104 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.871989 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8bba8d-4e50-44c1-bd3e-66a87c8be330" containerName="revoke-keys" Nov 01 16:03:04.872168 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.872113 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8bba8d-4e50-44c1-bd3e-66a87c8be330" containerName="revoke-keys" Nov 01 16:03:04.874896 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.874875 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:04.932097 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.932062 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48"] Nov 01 16:03:04.977859 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.977829 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:03:04.977859 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:04.977852 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:03:05.055918 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.055877 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mdx\" (UniqueName: \"kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.056112 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.056023 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.056112 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.056088 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.157664 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.157575 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mdx\" (UniqueName: \"kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.157849 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.157682 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.157849 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.157709 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.158378 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.158348 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.160076 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.160058 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.214290 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.214259 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mdx\" (UniqueName: \"kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx\") pod \"maas-api-revoke-sub-e2fc9c5b-jpj48\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.492865 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.492781 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:05.866629 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:05.866600 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48"] Nov 01 16:03:05.867825 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:03:05.867800 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf9de7b_6056_488b_9c99_0066f4f62d2d.slice/crio-a77242f5475a20257d25416ee002d5d6e8bce6c39a6803eee443847b834d3002 WatchSource:0}: Error finding container a77242f5475a20257d25416ee002d5d6e8bce6c39a6803eee443847b834d3002: Status 404 returned error can't find the container with id a77242f5475a20257d25416ee002d5d6e8bce6c39a6803eee443847b834d3002 Nov 01 16:03:06.670441 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:06.670411 5473 generic.go:356] "Generic (PLEG): container finished" podID="6bf9de7b-6056-488b-9c99-0066f4f62d2d" containerID="658cd051e93fd479e7ce12d167e8f2703311880da6fc51b6b86219c02d32cb25" exitCode=0 Nov 01 16:03:06.670827 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:06.670503 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" event={"ID":"6bf9de7b-6056-488b-9c99-0066f4f62d2d","Type":"ContainerDied","Data":"658cd051e93fd479e7ce12d167e8f2703311880da6fc51b6b86219c02d32cb25"} Nov 01 16:03:06.670827 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:06.670558 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" event={"ID":"6bf9de7b-6056-488b-9c99-0066f4f62d2d","Type":"ContainerStarted","Data":"a77242f5475a20257d25416ee002d5d6e8bce6c39a6803eee443847b834d3002"} Nov 01 16:03:07.817655 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.817632 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:07.988166 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.988077 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca\") pod \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " Nov 01 16:03:07.988166 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.988123 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx\" (UniqueName: \"kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx\") pod \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " Nov 01 16:03:07.988467 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.988264 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca\") pod \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\" (UID: \"6bf9de7b-6056-488b-9c99-0066f4f62d2d\") " Nov 01 16:03:07.988635 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.988610 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca" pod "6bf9de7b-6056-488b-9c99-0066f4f62d2d" (UID: "6bf9de7b-6056-488b-9c99-0066f4f62d2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:07.990396 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.990364 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca" pod "6bf9de7b-6056-488b-9c99-0066f4f62d2d" (UID: "6bf9de7b-6056-488b-9c99-0066f4f62d2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:07.990518 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:07.990475 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx" pod "6bf9de7b-6056-488b-9c99-0066f4f62d2d" (UID: "6bf9de7b-6056-488b-9c99-0066f4f62d2d"). InnerVolumeSpecName "kube-api-access-q8mdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:08.089334 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:08.089299 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:08.089334 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:08.089329 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bf9de7b-6056-488b-9c99-0066f4f62d2d-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:08.089334 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:08.089338 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mdx\" (UniqueName: \"kubernetes.io/projected/6bf9de7b-6056-488b-9c99-0066f4f62d2d-kube-api-access-q8mdx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:08.683612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:08.683577 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" event={"ID":"6bf9de7b-6056-488b-9c99-0066f4f62d2d","Type":"ContainerDied","Data":"a77242f5475a20257d25416ee002d5d6e8bce6c39a6803eee443847b834d3002"} Nov 01 16:03:08.683612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:08.683608 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48" Nov 01 16:03:08.683840 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:08.683633 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a77242f5475a20257d25416ee002d5d6e8bce6c39a6803eee443847b834d3002" Nov 01 16:03:15.625113 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.625075 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz"] Nov 01 16:03:15.625634 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.625502 5473 cpu_manager.go:414] "containerMap: removing container" podUID="6bf9de7b-6056-488b-9c99-0066f4f62d2d" containerName="revoke-keys" Nov 01 16:03:15.625634 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.625514 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf9de7b-6056-488b-9c99-0066f4f62d2d" containerName="revoke-keys" Nov 01 16:03:15.625634 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.625584 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf9de7b-6056-488b-9c99-0066f4f62d2d" containerName="revoke-keys" Nov 01 16:03:15.629086 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.629061 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.656021 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.655988 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.656146 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.656111 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7pm\" (UniqueName: \"kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.656196 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.656156 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.681937 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.681911 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:03:15.731962 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.731929 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:03:15.734988 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.734964 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz"] Nov 01 16:03:15.757027 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.757005 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.757108 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.757073 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7pm\" (UniqueName: \"kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.757182 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.757124 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.757810 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.757789 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.759378 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.759357 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.814063 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.814029 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7pm\" (UniqueName: \"kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm\") pod \"maas-api-revoke-sub-5a29efb3-pfzpz\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:15.942548 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:15.942406 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:16.313704 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:16.313675 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz"] Nov 01 16:03:16.315182 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:03:16.315160 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27b5314_c52b_4a79_9746_d88ad733fce1.slice/crio-a3ab995a04d6ad3a7ba5ae5cdd5482b057ef21d3e6f5d7962e1997707df3e096 WatchSource:0}: Error finding container a3ab995a04d6ad3a7ba5ae5cdd5482b057ef21d3e6f5d7962e1997707df3e096: Status 404 returned error can't find the container with id a3ab995a04d6ad3a7ba5ae5cdd5482b057ef21d3e6f5d7962e1997707df3e096 Nov 01 16:03:16.729814 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:16.729780 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" event={"ID":"d27b5314-c52b-4a79-9746-d88ad733fce1","Type":"ContainerStarted","Data":"4e3e540e6a0ab9cf5873b2d57dc414aadfc2a7a8e5708978154e41ff74c8ecbe"} Nov 01 16:03:16.730174 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:16.729832 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" event={"ID":"d27b5314-c52b-4a79-9746-d88ad733fce1","Type":"ContainerStarted","Data":"a3ab995a04d6ad3a7ba5ae5cdd5482b057ef21d3e6f5d7962e1997707df3e096"} Nov 01 16:03:16.844774 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:16.844745 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:16.844943 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:16.844804 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" podStartSLOduration=1.844788237 podStartE2EDuration="1.844788237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:16.841847093 +0000 UTC m=+6335.899998942" watchObservedRunningTime="2026-11-01 16:03:16.844788237 +0000 UTC m=+6335.902940092" Nov 01 16:03:17.737824 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:17.737789 5473 generic.go:356] "Generic (PLEG): container finished" podID="d27b5314-c52b-4a79-9746-d88ad733fce1" containerID="4e3e540e6a0ab9cf5873b2d57dc414aadfc2a7a8e5708978154e41ff74c8ecbe" exitCode=0 Nov 01 16:03:17.738263 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:17.737882 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" event={"ID":"d27b5314-c52b-4a79-9746-d88ad733fce1","Type":"ContainerDied","Data":"4e3e540e6a0ab9cf5873b2d57dc414aadfc2a7a8e5708978154e41ff74c8ecbe"} Nov 01 16:03:19.517745 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.517387 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:19.630424 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.629506 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca\") pod \"d27b5314-c52b-4a79-9746-d88ad733fce1\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " Nov 01 16:03:19.630424 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.629615 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm\" (UniqueName: \"kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm\") pod \"d27b5314-c52b-4a79-9746-d88ad733fce1\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " Nov 01 16:03:19.630424 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.629697 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca\") pod \"d27b5314-c52b-4a79-9746-d88ad733fce1\" (UID: \"d27b5314-c52b-4a79-9746-d88ad733fce1\") " Nov 01 16:03:19.631153 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.631126 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca" pod "d27b5314-c52b-4a79-9746-d88ad733fce1" (UID: "d27b5314-c52b-4a79-9746-d88ad733fce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:19.662494 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.662386 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm" pod "d27b5314-c52b-4a79-9746-d88ad733fce1" (UID: "d27b5314-c52b-4a79-9746-d88ad733fce1"). InnerVolumeSpecName "kube-api-access-ft7pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:19.671418 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.671372 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca" pod "d27b5314-c52b-4a79-9746-d88ad733fce1" (UID: "d27b5314-c52b-4a79-9746-d88ad733fce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:19.734948 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.734867 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:19.734948 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.734906 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7pm\" (UniqueName: \"kubernetes.io/projected/d27b5314-c52b-4a79-9746-d88ad733fce1-kube-api-access-ft7pm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:19.734948 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.734922 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d27b5314-c52b-4a79-9746-d88ad733fce1-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:19.834239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.813367 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" event={"ID":"d27b5314-c52b-4a79-9746-d88ad733fce1","Type":"ContainerDied","Data":"a3ab995a04d6ad3a7ba5ae5cdd5482b057ef21d3e6f5d7962e1997707df3e096"} Nov 01 16:03:19.834239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.813436 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ab995a04d6ad3a7ba5ae5cdd5482b057ef21d3e6f5d7962e1997707df3e096" Nov 01 16:03:19.834239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:19.813510 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz" Nov 01 16:03:30.631075 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:30.631038 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:30.797594 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:30.797563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:31.106291 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:03:31.106252 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:03:31.982587 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:31.982552 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd"] Nov 01 16:03:32.031995 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.031961 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w88pd"] Nov 01 16:03:32.094453 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.094417 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf"] Nov 01 16:03:32.094833 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.094820 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d27b5314-c52b-4a79-9746-d88ad733fce1" containerName="revoke-keys" Nov 01 16:03:32.094894 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.094834 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27b5314-c52b-4a79-9746-d88ad733fce1" containerName="revoke-keys" Nov 01 16:03:32.094933 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.094899 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27b5314-c52b-4a79-9746-d88ad733fce1" containerName="revoke-keys" Nov 01 16:03:32.098114 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.098097 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.151108 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.151079 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:03:32.151339 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.151242 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:03:32.154060 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.154037 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf"] Nov 01 16:03:32.175653 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.175618 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.175792 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.175735 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8vrb\" (UniqueName: \"kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.175866 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.175820 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.277300 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.277265 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8vrb\" (UniqueName: \"kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.277752 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.277317 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.277752 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.277451 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.278165 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.278140 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.279647 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.279626 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.333699 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.333672 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8vrb\" (UniqueName: \"kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb\") pod \"maas-api-revoke-sub-3d4156f1-wjvxf\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.411205 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.411177 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:32.782796 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.782768 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf"] Nov 01 16:03:32.789189 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:03:32.789157 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f04e39_9fad_411b_bf2b_54c733930b6d.slice/crio-b34d7cde5bde74a6744fc6dc93877982c734f8ccc712a606b3d8bb0336f761ea WatchSource:0}: Error finding container b34d7cde5bde74a6744fc6dc93877982c734f8ccc712a606b3d8bb0336f761ea: Status 404 returned error can't find the container with id b34d7cde5bde74a6744fc6dc93877982c734f8ccc712a606b3d8bb0336f761ea Nov 01 16:03:32.898401 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:32.898366 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" event={"ID":"93f04e39-9fad-411b-bf2b-54c733930b6d","Type":"ContainerStarted","Data":"b34d7cde5bde74a6744fc6dc93877982c734f8ccc712a606b3d8bb0336f761ea"} Nov 01 16:03:33.080195 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:33.080123 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8bba8d-4e50-44c1-bd3e-66a87c8be330" path="/var/lib/kubelet/pods/0e8bba8d-4e50-44c1-bd3e-66a87c8be330/volumes" Nov 01 16:03:33.907087 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:33.907055 5473 generic.go:356] "Generic (PLEG): container finished" podID="93f04e39-9fad-411b-bf2b-54c733930b6d" containerID="831d2e538b1a9151998c1cfe29ad3d73c17c0515da995a32f8ea6b1441443d1e" exitCode=0 Nov 01 16:03:33.907289 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:33.907138 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" event={"ID":"93f04e39-9fad-411b-bf2b-54c733930b6d","Type":"ContainerDied","Data":"831d2e538b1a9151998c1cfe29ad3d73c17c0515da995a32f8ea6b1441443d1e"} Nov 01 16:03:35.044396 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.044367 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:35.210282 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.210197 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca\") pod \"93f04e39-9fad-411b-bf2b-54c733930b6d\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " Nov 01 16:03:35.210282 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.210258 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca\") pod \"93f04e39-9fad-411b-bf2b-54c733930b6d\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " Nov 01 16:03:35.210486 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.210349 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb\" (UniqueName: \"kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb\") pod \"93f04e39-9fad-411b-bf2b-54c733930b6d\" (UID: \"93f04e39-9fad-411b-bf2b-54c733930b6d\") " Nov 01 16:03:35.210637 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.210611 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca" pod "93f04e39-9fad-411b-bf2b-54c733930b6d" (UID: "93f04e39-9fad-411b-bf2b-54c733930b6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:35.212423 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.212401 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca" pod "93f04e39-9fad-411b-bf2b-54c733930b6d" (UID: "93f04e39-9fad-411b-bf2b-54c733930b6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:35.212536 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.212485 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb" pod "93f04e39-9fad-411b-bf2b-54c733930b6d" (UID: "93f04e39-9fad-411b-bf2b-54c733930b6d"). InnerVolumeSpecName "kube-api-access-v8vrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:35.312077 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.312048 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8vrb\" (UniqueName: \"kubernetes.io/projected/93f04e39-9fad-411b-bf2b-54c733930b6d-kube-api-access-v8vrb\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:35.312077 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.312077 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:35.312280 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.312088 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04e39-9fad-411b-bf2b-54c733930b6d-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:35.920838 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.920805 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" Nov 01 16:03:35.921017 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.920834 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf" event={"ID":"93f04e39-9fad-411b-bf2b-54c733930b6d","Type":"ContainerDied","Data":"b34d7cde5bde74a6744fc6dc93877982c734f8ccc712a606b3d8bb0336f761ea"} Nov 01 16:03:35.921017 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:35.920928 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b34d7cde5bde74a6744fc6dc93877982c734f8ccc712a606b3d8bb0336f761ea" Nov 01 16:03:41.047352 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:41.047321 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd"] Nov 01 16:03:41.097643 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:41.097613 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd"] Nov 01 16:03:41.128892 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:03:41.128854 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-e2547622-jcvbd\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="f80574a8-8ae6-46de-8964-804ef1ff69bc" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-jcvbd" Nov 01 16:03:43.080504 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:43.080470 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f80574a8-8ae6-46de-8964-804ef1ff69bc" path="/var/lib/kubelet/pods/f80574a8-8ae6-46de-8964-804ef1ff69bc/volumes" Nov 01 16:03:47.972758 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:47.972725 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:48.531747 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:48.531708 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:48.731954 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:48.731925 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:52.317175 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:52.317140 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:52.480301 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:52.480262 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:52.648183 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:52.648101 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.490316 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.490284 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.637999 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.637965 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.829599 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.829568 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr"] Nov 01 16:03:55.829974 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.829962 5473 cpu_manager.go:414] "containerMap: removing container" podUID="93f04e39-9fad-411b-bf2b-54c733930b6d" containerName="revoke-keys" Nov 01 16:03:55.830023 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.829975 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f04e39-9fad-411b-bf2b-54c733930b6d" containerName="revoke-keys" Nov 01 16:03:55.830081 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.830072 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f04e39-9fad-411b-bf2b-54c733930b6d" containerName="revoke-keys" Nov 01 16:03:55.834631 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.834614 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:55.889487 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.889459 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr"] Nov 01 16:03:55.901479 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.901440 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5b6\" (UniqueName: \"kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6\") pod \"payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:55.901642 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.901569 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:55.936334 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.936300 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9ebec3\"" Nov 01 16:03:55.936622 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.936492 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c699d060-dockercfg-ph7pl\"" Nov 01 16:03:55.989046 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.989014 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w"] Nov 01 16:03:55.992714 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:55.992696 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.002119 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.002091 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5b6\" (UniqueName: \"kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6\") pod \"payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:56.002305 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.002174 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:56.002878 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.002859 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:56.047397 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.047370 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w"] Nov 01 16:03:56.058944 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.058919 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5b6\" (UniqueName: \"kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6\") pod \"payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:56.102994 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.102923 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghkf\" (UniqueName: \"kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf\") pod \"payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.103145 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.103080 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.147541 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.147511 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:56.205233 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.203811 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghkf\" (UniqueName: \"kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf\") pod \"payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.205233 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.203911 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.205233 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.204769 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.260750 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.260715 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghkf\" (UniqueName: \"kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf\") pod \"payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.307791 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.307759 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:56.520343 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.520316 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr"] Nov 01 16:03:56.522404 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:03:56.522378 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932c8ea9_5e6a_4c8e_90f8_23b610643de3.slice/crio-584d6326ed243741bb405445e93ab5b3f3dec81eac9a7f1d0b4a1990156bdfc3 WatchSource:0}: Error finding container 584d6326ed243741bb405445e93ab5b3f3dec81eac9a7f1d0b4a1990156bdfc3: Status 404 returned error can't find the container with id 584d6326ed243741bb405445e93ab5b3f3dec81eac9a7f1d0b4a1990156bdfc3 Nov 01 16:03:56.678320 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:56.678289 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w"] Nov 01 16:03:56.679157 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:03:56.679133 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb03108a_7313_44c7_ad27_ba203bc4c65a.slice/crio-664bda85077cab90eaafe8a281eb5205ce1c0214495509b6077e52fcf0aec3e5 WatchSource:0}: Error finding container 664bda85077cab90eaafe8a281eb5205ce1c0214495509b6077e52fcf0aec3e5: Status 404 returned error can't find the container with id 664bda85077cab90eaafe8a281eb5205ce1c0214495509b6077e52fcf0aec3e5 Nov 01 16:03:57.041407 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.041372 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" event={"ID":"bb03108a-7313-44c7-ad27-ba203bc4c65a","Type":"ContainerStarted","Data":"013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1"} Nov 01 16:03:57.041597 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.041429 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:03:57.041597 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.041445 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" event={"ID":"bb03108a-7313-44c7-ad27-ba203bc4c65a","Type":"ContainerStarted","Data":"664bda85077cab90eaafe8a281eb5205ce1c0214495509b6077e52fcf0aec3e5"} Nov 01 16:03:57.043097 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.043072 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" event={"ID":"932c8ea9-5e6a-4c8e-90f8-23b610643de3","Type":"ContainerStarted","Data":"f8586e5b313e674bed0f767a2ca3c9408c8ba92f1d523169bbda3de4e0516db2"} Nov 01 16:03:57.043247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.043119 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" event={"ID":"932c8ea9-5e6a-4c8e-90f8-23b610643de3","Type":"ContainerStarted","Data":"584d6326ed243741bb405445e93ab5b3f3dec81eac9a7f1d0b4a1990156bdfc3"} Nov 01 16:03:57.043247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.043154 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:03:57.157122 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.157074 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" podStartSLOduration=2.1570600779999998 podStartE2EDuration="2.157060078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:57.154202663 +0000 UTC m=+6376.212354514" watchObservedRunningTime="2026-11-01 16:03:57.157060078 +0000 UTC m=+6376.215211925" Nov 01 16:03:57.268946 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:57.268887 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" podStartSLOduration=2.268869432 podStartE2EDuration="2.268869432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:57.266644157 +0000 UTC m=+6376.324796006" watchObservedRunningTime="2026-11-01 16:03:57.268869432 +0000 UTC m=+6376.327021280" Nov 01 16:03:58.052361 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:58.052331 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:58.205765 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:58.205731 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:58.369995 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:03:58.369903 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.107240 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:04:01.107195 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:04:02.596284 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:02.596240 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:07.117839 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:07.117805 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:08.051770 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:08.051741 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:04:08.051965 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:08.051808 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:04:11.046555 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:11.046514 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2"] Nov 01 16:04:11.096468 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:11.096432 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2"] Nov 01 16:04:11.128870 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:04:11.128833 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-c1e271a9-vpxw2\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="27e318ec-c81a-40af-a9ee-327e1573e272" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vpxw2" Nov 01 16:04:11.609614 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:11.609581 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:13.080339 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:13.080310 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e318ec-c81a-40af-a9ee-327e1573e272" path="/var/lib/kubelet/pods/27e318ec-c81a-40af-a9ee-327e1573e272/volumes" Nov 01 16:04:16.322673 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:16.322634 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:17.080713 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:17.080681 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:17.243513 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:17.243477 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:17.617864 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:17.617829 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:17.891551 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:17.891462 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:20.816438 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:20.816362 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:21.154904 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:21.154828 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:25.300463 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:25.300427 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:29.815139 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:29.815100 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:31.107335 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:04:31.107297 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:04:34.181781 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:34.181746 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:37.046424 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:37.046380 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267"] Nov 01 16:04:37.095903 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:37.095870 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-5w267"] Nov 01 16:04:38.589573 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:38.589542 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.085127 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:39.085065 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2576f13e-473f-473c-9496-1e0ab018edd3" path="/var/lib/kubelet/pods/2576f13e-473f-473c-9496-1e0ab018edd3/volumes" Nov 01 16:04:43.016721 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:43.016682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:45.750589 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:45.750556 5473 scope.go:122] "RemoveContainer" containerID="65f9036930ff8024afb0c9fa1836b7aaac96d2e0e9c8727367533e0c87bb40fe" Nov 01 16:04:47.495745 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:47.495708 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:49.048186 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:49.048140 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz"] Nov 01 16:04:49.097743 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:49.097713 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q6kjz"] Nov 01 16:04:51.081037 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:51.081003 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c239fcb6-18d5-49aa-ba69-92a3449761b8" path="/var/lib/kubelet/pods/c239fcb6-18d5-49aa-ba69-92a3449761b8/volumes" Nov 01 16:04:52.008972 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:52.008942 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:56.413014 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:04:56.412984 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:00.830688 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:00.830655 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:01.107634 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:05:01.107547 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:05:04.903716 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:04.903680 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr"] Nov 01 16:05:04.907242 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:04.907206 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:04.963488 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:04.963451 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr"] Nov 01 16:05:04.987992 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:04.987966 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8j7\" (UniqueName: \"kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:04.988166 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:04.988087 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:04.988166 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:04.988130 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.009731 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.009705 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:05:05.009731 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.009705 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:05:05.089554 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.089517 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.089554 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.089562 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.089776 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.089608 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8j7\" (UniqueName: \"kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.090300 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.090275 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.092001 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.091963 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.146658 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.146632 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8j7\" (UniqueName: \"kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7\") pod \"maas-api-revoke-sub-a93ac944-dcptr\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.219896 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.219805 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:05.220269 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.220246 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:05.595640 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:05.595611 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr"] Nov 01 16:05:05.597601 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:05:05.597574 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05024251_322e_4d9e_867a_075108dc785a.slice/crio-57691e4f35f7449b0975d19ec58bf25aecc2db2f2b4a540286c98a1d02c078ed WatchSource:0}: Error finding container 57691e4f35f7449b0975d19ec58bf25aecc2db2f2b4a540286c98a1d02c078ed: Status 404 returned error can't find the container with id 57691e4f35f7449b0975d19ec58bf25aecc2db2f2b4a540286c98a1d02c078ed Nov 01 16:05:06.444775 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:06.444740 5473 generic.go:356] "Generic (PLEG): container finished" podID="05024251-322e-4d9e-867a-075108dc785a" containerID="3cfb592f0bba2c01c4de87523756dcb9a42141848a1bf798e58a3369340a9695" exitCode=0 Nov 01 16:05:06.445182 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:06.444784 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" event={"ID":"05024251-322e-4d9e-867a-075108dc785a","Type":"ContainerDied","Data":"3cfb592f0bba2c01c4de87523756dcb9a42141848a1bf798e58a3369340a9695"} Nov 01 16:05:06.445182 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:06.444830 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" event={"ID":"05024251-322e-4d9e-867a-075108dc785a","Type":"ContainerStarted","Data":"57691e4f35f7449b0975d19ec58bf25aecc2db2f2b4a540286c98a1d02c078ed"} Nov 01 16:05:07.585954 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.585927 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:07.615255 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.615230 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7\" (UniqueName: \"kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7\") pod \"05024251-322e-4d9e-867a-075108dc785a\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " Nov 01 16:05:07.615380 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.615296 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca\") pod \"05024251-322e-4d9e-867a-075108dc785a\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " Nov 01 16:05:07.615380 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.615344 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca\") pod \"05024251-322e-4d9e-867a-075108dc785a\" (UID: \"05024251-322e-4d9e-867a-075108dc785a\") " Nov 01 16:05:07.615646 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.615625 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca" pod "05024251-322e-4d9e-867a-075108dc785a" (UID: "05024251-322e-4d9e-867a-075108dc785a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:07.617287 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.617262 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7" pod "05024251-322e-4d9e-867a-075108dc785a" (UID: "05024251-322e-4d9e-867a-075108dc785a"). InnerVolumeSpecName "kube-api-access-jg8j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:07.617402 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.617311 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca" pod "05024251-322e-4d9e-867a-075108dc785a" (UID: "05024251-322e-4d9e-867a-075108dc785a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:07.716601 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.716527 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05024251-322e-4d9e-867a-075108dc785a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:07.716601 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.716554 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8j7\" (UniqueName: \"kubernetes.io/projected/05024251-322e-4d9e-867a-075108dc785a-kube-api-access-jg8j7\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:07.716601 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:07.716564 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05024251-322e-4d9e-867a-075108dc785a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:08.457452 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:08.457416 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" Nov 01 16:05:08.457647 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:08.457449 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr" event={"ID":"05024251-322e-4d9e-867a-075108dc785a","Type":"ContainerDied","Data":"57691e4f35f7449b0975d19ec58bf25aecc2db2f2b4a540286c98a1d02c078ed"} Nov 01 16:05:08.457647 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:08.457509 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57691e4f35f7449b0975d19ec58bf25aecc2db2f2b4a540286c98a1d02c078ed" Nov 01 16:05:09.689674 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:09.689646 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:14.095027 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:14.094990 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:16.460589 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:16.460536 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:16.896150 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:16.896110 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:17.421165 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:17.421135 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:18.499809 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:18.499774 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:20.378922 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:20.378889 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:21.490576 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:21.490539 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.060611 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:22.060572 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w"] Nov 01 16:05:22.165853 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:22.165824 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr"] Nov 01 16:05:22.436665 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:22.436585 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.594749 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:22.594717 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:23.054144 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:23.054067 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:23.061226 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:23.061189 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd"] Nov 01 16:05:23.064355 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:23.064331 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-v25qd"] Nov 01 16:05:23.080069 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:23.080045 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dce9d9a-c09e-4f4b-9494-0899dddb6c47" path="/var/lib/kubelet/pods/0dce9d9a-c09e-4f4b-9494-0899dddb6c47/volumes" Nov 01 16:05:23.208163 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:23.208133 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:27.046426 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:27.046397 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:27.061268 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:27.061204 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerName="payload-processing" containerID="cri-o://013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1" gracePeriod=25 Nov 01 16:05:27.539669 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:27.539603 5473 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerName="payload-pre-processing" containerID="cri-o://f8586e5b313e674bed0f767a2ca3c9408c8ba92f1d523169bbda3de4e0516db2" gracePeriod=25 Nov 01 16:05:28.050412 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:28.050374 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:05:28.050412 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:28.050401 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:05:28.744761 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:28.744727 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:28.914083 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:28.914047 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:31.106551 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:05:31.106510 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:05:34.053446 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:34.053412 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw"] Nov 01 16:05:34.103210 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:34.103177 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ndhlw"] Nov 01 16:05:35.080228 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:35.080183 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b36878-ec14-4546-be91-f8548400db00" path="/var/lib/kubelet/pods/67b36878-ec14-4546-be91-f8548400db00/volumes" Nov 01 16:05:38.051392 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:38.051351 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:05:38.051392 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:38.051368 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:05:45.773923 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:45.773892 5473 scope.go:122] "RemoveContainer" containerID="4d7c3cf8d792e0a5a19a224cf079f3d0767501e96e183f82426b75154c7e717f" Nov 01 16:05:45.785920 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:45.785892 5473 scope.go:122] "RemoveContainer" containerID="e3fe04c63c4c374c20512b2561493ab7c730eb5e3bfe960405cdc68e539f7aaf" Nov 01 16:05:47.319625 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.319566 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:05:47.412083 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.412056 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf\" (UniqueName: \"kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf\") pod \"bb03108a-7313-44c7-ad27-ba203bc4c65a\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " Nov 01 16:05:47.412309 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.412198 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume\") pod \"bb03108a-7313-44c7-ad27-ba203bc4c65a\" (UID: \"bb03108a-7313-44c7-ad27-ba203bc4c65a\") " Nov 01 16:05:47.412578 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.412556 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume" pod "bb03108a-7313-44c7-ad27-ba203bc4c65a" (UID: "bb03108a-7313-44c7-ad27-ba203bc4c65a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:47.414429 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.414411 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf" pod "bb03108a-7313-44c7-ad27-ba203bc4c65a" (UID: "bb03108a-7313-44c7-ad27-ba203bc4c65a"). InnerVolumeSpecName "kube-api-access-sghkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:47.514044 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.514009 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb03108a-7313-44c7-ad27-ba203bc4c65a-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:47.514044 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.514038 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghkf\" (UniqueName: \"kubernetes.io/projected/bb03108a-7313-44c7-ad27-ba203bc4c65a-kube-api-access-sghkf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:47.689128 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.689095 5473 generic.go:356] "Generic (PLEG): container finished" podID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerID="f8586e5b313e674bed0f767a2ca3c9408c8ba92f1d523169bbda3de4e0516db2" exitCode=0 Nov 01 16:05:47.689317 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.689170 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" event={"ID":"932c8ea9-5e6a-4c8e-90f8-23b610643de3","Type":"ContainerDied","Data":"f8586e5b313e674bed0f767a2ca3c9408c8ba92f1d523169bbda3de4e0516db2"} Nov 01 16:05:47.690543 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.690521 5473 generic.go:356] "Generic (PLEG): container finished" podID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerID="013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1" exitCode=0 Nov 01 16:05:47.690675 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.690556 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" event={"ID":"bb03108a-7313-44c7-ad27-ba203bc4c65a","Type":"ContainerDied","Data":"013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1"} Nov 01 16:05:47.690675 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.690615 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" event={"ID":"bb03108a-7313-44c7-ad27-ba203bc4c65a","Type":"ContainerDied","Data":"664bda85077cab90eaafe8a281eb5205ce1c0214495509b6077e52fcf0aec3e5"} Nov 01 16:05:47.690675 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.690572 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w" Nov 01 16:05:47.690675 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.690651 5473 scope.go:122] "RemoveContainer" containerID="013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1" Nov 01 16:05:47.701638 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.701619 5473 scope.go:122] "RemoveContainer" containerID="013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1" Nov 01 16:05:47.701925 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:05:47.701907 5473 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1\": container with ID starting with 013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1 not found: ID does not exist" containerID="013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1" Nov 01 16:05:47.701986 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.701936 5473 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1"} err="failed to get container status \"013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1\": rpc error: code = NotFound desc = could not find container \"013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1\": container with ID starting with 013b84667940664641dde637f085b1b71df78068ac51ffe18fa026db1fe3f4e1 not found: ID does not exist" Nov 01 16:05:47.784674 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.784649 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:05:47.816817 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.816787 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume\") pod \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " Nov 01 16:05:47.816954 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.816924 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6\" (UniqueName: \"kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6\") pod \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\" (UID: \"932c8ea9-5e6a-4c8e-90f8-23b610643de3\") " Nov 01 16:05:47.817130 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.817109 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume" pod "932c8ea9-5e6a-4c8e-90f8-23b610643de3" (UID: "932c8ea9-5e6a-4c8e-90f8-23b610643de3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:47.818732 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.818706 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6" pod "932c8ea9-5e6a-4c8e-90f8-23b610643de3" (UID: "932c8ea9-5e6a-4c8e-90f8-23b610643de3"). InnerVolumeSpecName "kube-api-access-xd5b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:47.856431 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.856403 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w"] Nov 01 16:05:47.906347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.906287 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9ebec3-5bf56b47424p9w"] Nov 01 16:05:47.917609 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.917578 5473 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932c8ea9-5e6a-4c8e-90f8-23b610643de3-plugins-config-volume\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:47.917609 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:47.917606 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5b6\" (UniqueName: \"kubernetes.io/projected/932c8ea9-5e6a-4c8e-90f8-23b610643de3-kube-api-access-xd5b6\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:48.697766 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:48.697739 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" Nov 01 16:05:48.697766 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:48.697754 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr" event={"ID":"932c8ea9-5e6a-4c8e-90f8-23b610643de3","Type":"ContainerDied","Data":"584d6326ed243741bb405445e93ab5b3f3dec81eac9a7f1d0b4a1990156bdfc3"} Nov 01 16:05:48.698330 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:48.697809 5473 scope.go:122] "RemoveContainer" containerID="f8586e5b313e674bed0f767a2ca3c9408c8ba92f1d523169bbda3de4e0516db2" Nov 01 16:05:48.863940 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:48.863903 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr"] Nov 01 16:05:48.867611 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:48.867578 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-9ebec3-8cdf425wmr"] Nov 01 16:05:49.080546 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:49.080514 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" path="/var/lib/kubelet/pods/932c8ea9-5e6a-4c8e-90f8-23b610643de3/volumes" Nov 01 16:05:49.080894 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:49.080881 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" path="/var/lib/kubelet/pods/bb03108a-7313-44c7-ad27-ba203bc4c65a/volumes" Nov 01 16:05:58.193068 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:58.192977 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:58.398404 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:58.398369 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:58.701804 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:58.701766 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:58.883355 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:05:58.883323 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:00.129347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:00.129306 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:00.466191 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:00.466117 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:01.065045 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:01.065012 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:01.106813 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:06:01.106780 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:06:02.526894 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.526855 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf"] Nov 01 16:06:02.576720 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.576688 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjvxf"] Nov 01 16:06:02.640521 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640491 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw"] Nov 01 16:06:02.640886 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640873 5473 cpu_manager.go:414] "containerMap: removing container" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerName="payload-processing" Nov 01 16:06:02.640938 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640888 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerName="payload-processing" Nov 01 16:06:02.640938 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640907 5473 cpu_manager.go:414] "containerMap: removing container" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerName="payload-pre-processing" Nov 01 16:06:02.640938 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640912 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerName="payload-pre-processing" Nov 01 16:06:02.640938 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640920 5473 cpu_manager.go:414] "containerMap: removing container" podUID="05024251-322e-4d9e-867a-075108dc785a" containerName="revoke-keys" Nov 01 16:06:02.640938 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.640925 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05024251-322e-4d9e-867a-075108dc785a" containerName="revoke-keys" Nov 01 16:06:02.641105 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.641003 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb03108a-7313-44c7-ad27-ba203bc4c65a" containerName="payload-processing" Nov 01 16:06:02.641105 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.641013 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="932c8ea9-5e6a-4c8e-90f8-23b610643de3" containerName="payload-pre-processing" Nov 01 16:06:02.641105 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.641019 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="05024251-322e-4d9e-867a-075108dc785a" containerName="revoke-keys" Nov 01 16:06:02.645463 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.645446 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.698588 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.698557 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:06:02.699395 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.699372 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:06:02.700783 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.700761 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw"] Nov 01 16:06:02.764156 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.764115 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.764425 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.764239 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnh49\" (UniqueName: \"kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.764425 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.764340 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.866027 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.865936 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnh49\" (UniqueName: \"kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.866243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.866029 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.866243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.866075 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.866691 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.866667 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.868315 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.868295 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.922485 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.922458 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnh49\" (UniqueName: \"kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49\") pod \"maas-api-revoke-sub-3d4156f1-btgcw\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:02.958456 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:02.958429 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:03.080651 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:03.080620 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f04e39-9fad-411b-bf2b-54c733930b6d" path="/var/lib/kubelet/pods/93f04e39-9fad-411b-bf2b-54c733930b6d/volumes" Nov 01 16:06:03.329698 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:03.329670 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw"] Nov 01 16:06:03.331777 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:06:03.331749 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bc5427_8176_480d_9dbd_d6f9b26a2079.slice/crio-39fc4effb05a988b4c06253ef7141f165abb24ba98e95728c23ea431934c4a7e WatchSource:0}: Error finding container 39fc4effb05a988b4c06253ef7141f165abb24ba98e95728c23ea431934c4a7e: Status 404 returned error can't find the container with id 39fc4effb05a988b4c06253ef7141f165abb24ba98e95728c23ea431934c4a7e Nov 01 16:06:03.786185 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:03.786138 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" event={"ID":"03bc5427-8176-480d-9dbd-d6f9b26a2079","Type":"ContainerStarted","Data":"91e5fb7da76f0e8ad2200127d173cb0d4d02a1edd3ef37809afb31b5997416e8"} Nov 01 16:06:03.786683 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:03.786200 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" event={"ID":"03bc5427-8176-480d-9dbd-d6f9b26a2079","Type":"ContainerStarted","Data":"39fc4effb05a988b4c06253ef7141f165abb24ba98e95728c23ea431934c4a7e"} Nov 01 16:06:03.901446 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:03.901341 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" podStartSLOduration=1.9013275680000001 podStartE2EDuration="1.901327568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:03.899704026 +0000 UTC m=+6502.957855876" watchObservedRunningTime="2026-11-01 16:06:03.901327568 +0000 UTC m=+6502.959479417" Nov 01 16:06:03.972076 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:03.972042 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:04.136591 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:04.136562 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:04.792925 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:04.792892 5473 generic.go:356] "Generic (PLEG): container finished" podID="03bc5427-8176-480d-9dbd-d6f9b26a2079" containerID="91e5fb7da76f0e8ad2200127d173cb0d4d02a1edd3ef37809afb31b5997416e8" exitCode=0 Nov 01 16:06:04.793381 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:04.792977 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" event={"ID":"03bc5427-8176-480d-9dbd-d6f9b26a2079","Type":"ContainerDied","Data":"91e5fb7da76f0e8ad2200127d173cb0d4d02a1edd3ef37809afb31b5997416e8"} Nov 01 16:06:05.922720 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:05.922695 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:06.099861 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.099768 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49\" (UniqueName: \"kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49\") pod \"03bc5427-8176-480d-9dbd-d6f9b26a2079\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " Nov 01 16:06:06.100048 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.099979 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca\") pod \"03bc5427-8176-480d-9dbd-d6f9b26a2079\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " Nov 01 16:06:06.100048 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.100025 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca\") pod \"03bc5427-8176-480d-9dbd-d6f9b26a2079\" (UID: \"03bc5427-8176-480d-9dbd-d6f9b26a2079\") " Nov 01 16:06:06.100447 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.100423 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca" pod "03bc5427-8176-480d-9dbd-d6f9b26a2079" (UID: "03bc5427-8176-480d-9dbd-d6f9b26a2079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:06.101969 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.101940 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca" pod "03bc5427-8176-480d-9dbd-d6f9b26a2079" (UID: "03bc5427-8176-480d-9dbd-d6f9b26a2079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:06:06.102077 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.101987 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49" pod "03bc5427-8176-480d-9dbd-d6f9b26a2079" (UID: "03bc5427-8176-480d-9dbd-d6f9b26a2079"). InnerVolumeSpecName "kube-api-access-qnh49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:06.201369 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.201334 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:06.201369 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.201360 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bc5427-8176-480d-9dbd-d6f9b26a2079-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:06.201369 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.201370 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnh49\" (UniqueName: \"kubernetes.io/projected/03bc5427-8176-480d-9dbd-d6f9b26a2079-kube-api-access-qnh49\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:06.805986 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.805945 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" Nov 01 16:06:06.806178 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.805980 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw" event={"ID":"03bc5427-8176-480d-9dbd-d6f9b26a2079","Type":"ContainerDied","Data":"39fc4effb05a988b4c06253ef7141f165abb24ba98e95728c23ea431934c4a7e"} Nov 01 16:06:06.806178 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:06.806037 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39fc4effb05a988b4c06253ef7141f165abb24ba98e95728c23ea431934c4a7e" Nov 01 16:06:08.046776 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:08.046740 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8"] Nov 01 16:06:08.096610 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:08.096580 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-rjmg8"] Nov 01 16:06:09.080287 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:09.080254 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39de830-66db-4557-8873-315787a975e9" path="/var/lib/kubelet/pods/f39de830-66db-4557-8873-315787a975e9/volumes" Nov 01 16:06:17.753724 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:17.753682 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:28.047224 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:28.047192 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54"] Nov 01 16:06:28.096841 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:28.096809 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cnw54"] Nov 01 16:06:29.080419 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:29.080383 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed3bc33-2476-44e8-9aba-15648f6f41fe" path="/var/lib/kubelet/pods/4ed3bc33-2476-44e8-9aba-15648f6f41fe/volumes" Nov 01 16:06:31.106360 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:06:31.106326 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:06:45.813571 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:45.813535 5473 scope.go:122] "RemoveContainer" containerID="6fca6d4f0a33444ef6c56f16f8912bc00c7794e7ca2c5940c7f5e5a1c7b826e4" Nov 01 16:06:45.825027 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:45.824999 5473 scope.go:122] "RemoveContainer" containerID="84d94ee22c4ba61a70a9a7b6a1757f1e6395d3590819ff3757f1ae811038164a" Nov 01 16:06:45.833167 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:45.833138 5473 scope.go:122] "RemoveContainer" containerID="6f3bba3f93dc6d5c5da752561aa9e9f21f8a7c3f596fd7c5f3d0a78bc344cd26" Nov 01 16:06:49.046570 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:49.046537 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4"] Nov 01 16:06:49.096024 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:49.095986 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-fk5w4"] Nov 01 16:06:51.080145 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:06:51.080114 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5655804-5f93-4dc7-af39-47624a7cf407" path="/var/lib/kubelet/pods/e5655804-5f93-4dc7-af39-47624a7cf407/volumes" Nov 01 16:07:01.106530 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:07:01.106493 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:07:05.047803 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:05.047767 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h"] Nov 01 16:07:05.097362 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:05.097329 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n5g9h"] Nov 01 16:07:07.080032 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:07.079981 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2970462c-af06-4313-8a24-bdc79fad1859" path="/var/lib/kubelet/pods/2970462c-af06-4313-8a24-bdc79fad1859/volumes" Nov 01 16:07:18.076598 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:18.076563 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:31.106894 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:07:31.106808 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:07:44.935478 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:44.935452 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:07:44.961046 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:44.961024 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:07:45.872974 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:45.872942 5473 scope.go:122] "RemoveContainer" containerID="5989a79b6abfb5582308126796ae71b41f836927ca65bf9deee055287e394eba" Nov 01 16:07:45.882134 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:45.882114 5473 scope.go:122] "RemoveContainer" containerID="f052bb7cc2a3d0bce109286daf26cac78d8c2220ee9adc3e0bac5e007c64a245" Nov 01 16:07:45.891932 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:45.891911 5473 scope.go:122] "RemoveContainer" containerID="ee79d07f11b54cb3ae15686c517a2f2f410c40bc6d0a09c9adffee2b0b41d17f" Nov 01 16:07:54.229575 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.229543 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk"] Nov 01 16:07:54.230151 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.230131 5473 cpu_manager.go:414] "containerMap: removing container" podUID="03bc5427-8176-480d-9dbd-d6f9b26a2079" containerName="revoke-keys" Nov 01 16:07:54.230255 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.230153 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bc5427-8176-480d-9dbd-d6f9b26a2079" containerName="revoke-keys" Nov 01 16:07:54.230323 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.230275 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bc5427-8176-480d-9dbd-d6f9b26a2079" containerName="revoke-keys" Nov 01 16:07:54.235261 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.235236 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.289476 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.289453 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk"] Nov 01 16:07:54.336879 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.336847 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:07:54.337034 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.336973 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:07:54.404970 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.404938 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.405123 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.405016 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smrqx\" (UniqueName: \"kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.405123 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.405048 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.506239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.506121 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrqx\" (UniqueName: \"kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.506239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.506165 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.506472 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.506311 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.506882 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.506863 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.508798 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.508773 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.563606 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.563575 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrqx\" (UniqueName: \"kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx\") pod \"maas-api-revoke-sub-d576e68b-vjqqk\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:54.847683 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:54.847650 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:55.218990 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:55.218956 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk"] Nov 01 16:07:55.220195 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:07:55.220167 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f1435e_1c24_429c_9147_c14332d7737c.slice/crio-a0151f58ff51b9a57610ebf0ef4a4035ced36b0fafd6b44d1cf11fe8127f1375 WatchSource:0}: Error finding container a0151f58ff51b9a57610ebf0ef4a4035ced36b0fafd6b44d1cf11fe8127f1375: Status 404 returned error can't find the container with id a0151f58ff51b9a57610ebf0ef4a4035ced36b0fafd6b44d1cf11fe8127f1375 Nov 01 16:07:55.434239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:55.434177 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" event={"ID":"47f1435e-1c24-429c-9147-c14332d7737c","Type":"ContainerStarted","Data":"a0151f58ff51b9a57610ebf0ef4a4035ced36b0fafd6b44d1cf11fe8127f1375"} Nov 01 16:07:56.440853 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:56.440817 5473 generic.go:356] "Generic (PLEG): container finished" podID="47f1435e-1c24-429c-9147-c14332d7737c" containerID="7bd575f943f422f2744013f9bd864817e328cee848c0c97e61c6701cb9985b58" exitCode=0 Nov 01 16:07:56.441270 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:56.440854 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" event={"ID":"47f1435e-1c24-429c-9147-c14332d7737c","Type":"ContainerDied","Data":"7bd575f943f422f2744013f9bd864817e328cee848c0c97e61c6701cb9985b58"} Nov 01 16:07:57.580449 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.580424 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:57.740063 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.739982 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca\") pod \"47f1435e-1c24-429c-9147-c14332d7737c\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " Nov 01 16:07:57.740272 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.740080 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx\" (UniqueName: \"kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx\") pod \"47f1435e-1c24-429c-9147-c14332d7737c\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " Nov 01 16:07:57.740272 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.740121 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca\") pod \"47f1435e-1c24-429c-9147-c14332d7737c\" (UID: \"47f1435e-1c24-429c-9147-c14332d7737c\") " Nov 01 16:07:57.740514 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.740491 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca" pod "47f1435e-1c24-429c-9147-c14332d7737c" (UID: "47f1435e-1c24-429c-9147-c14332d7737c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:07:57.742191 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.742169 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx" pod "47f1435e-1c24-429c-9147-c14332d7737c" (UID: "47f1435e-1c24-429c-9147-c14332d7737c"). InnerVolumeSpecName "kube-api-access-smrqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:07:57.742285 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.742270 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca" pod "47f1435e-1c24-429c-9147-c14332d7737c" (UID: "47f1435e-1c24-429c-9147-c14332d7737c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:07:57.841347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.841308 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:07:57.841347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.841339 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f1435e-1c24-429c-9147-c14332d7737c-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:07:57.841347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:57.841353 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smrqx\" (UniqueName: \"kubernetes.io/projected/47f1435e-1c24-429c-9147-c14332d7737c-kube-api-access-smrqx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:07:58.453136 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:58.453099 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" event={"ID":"47f1435e-1c24-429c-9147-c14332d7737c","Type":"ContainerDied","Data":"a0151f58ff51b9a57610ebf0ef4a4035ced36b0fafd6b44d1cf11fe8127f1375"} Nov 01 16:07:58.453136 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:58.453131 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk" Nov 01 16:07:58.453394 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:07:58.453181 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0151f58ff51b9a57610ebf0ef4a4035ced36b0fafd6b44d1cf11fe8127f1375" Nov 01 16:08:01.107139 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:08:01.107102 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:08:08.048432 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:08.048399 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48"] Nov 01 16:08:08.100414 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:08.100376 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-jpj48"] Nov 01 16:08:09.080129 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:09.080096 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf9de7b-6056-488b-9c99-0066f4f62d2d" path="/var/lib/kubelet/pods/6bf9de7b-6056-488b-9c99-0066f4f62d2d/volumes" Nov 01 16:08:15.026889 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.026854 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:15.195906 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.195871 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:15.369568 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.369482 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql"] Nov 01 16:08:15.369992 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.369972 5473 cpu_manager.go:414] "containerMap: removing container" podUID="47f1435e-1c24-429c-9147-c14332d7737c" containerName="revoke-keys" Nov 01 16:08:15.370098 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.369994 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f1435e-1c24-429c-9147-c14332d7737c" containerName="revoke-keys" Nov 01 16:08:15.370167 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.370141 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f1435e-1c24-429c-9147-c14332d7737c" containerName="revoke-keys" Nov 01 16:08:15.373386 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.373365 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.429561 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.429529 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql"] Nov 01 16:08:15.477031 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.476997 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:08:15.477031 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.477031 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:08:15.560406 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.560373 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.560582 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.560496 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6thc\" (UniqueName: \"kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.560582 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.560531 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.661383 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.661303 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6thc\" (UniqueName: \"kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.661383 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.661346 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.661647 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.661405 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.662005 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.661985 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.663719 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.663699 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.718093 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.718066 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6thc\" (UniqueName: \"kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc\") pod \"maas-api-revoke-sub-d2bc93bc-8gvql\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:15.986400 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:15.986322 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:16.359802 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:16.359777 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql"] Nov 01 16:08:16.361256 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:08:16.361229 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243fd851_1e2a_47a3_bd76_ba213767796a.slice/crio-3441a89c2c78769ebcbbb2ba6645d7c1b117f67302caedff39dd5308310cddda WatchSource:0}: Error finding container 3441a89c2c78769ebcbbb2ba6645d7c1b117f67302caedff39dd5308310cddda: Status 404 returned error can't find the container with id 3441a89c2c78769ebcbbb2ba6645d7c1b117f67302caedff39dd5308310cddda Nov 01 16:08:16.604071 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:16.604028 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" event={"ID":"243fd851-1e2a-47a3-bd76-ba213767796a","Type":"ContainerStarted","Data":"8d1b7609c28c6561f530b829ac5288eec0f8219eaec32672d8bca7cfdfb06877"} Nov 01 16:08:16.604244 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:16.604095 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" event={"ID":"243fd851-1e2a-47a3-bd76-ba213767796a","Type":"ContainerStarted","Data":"3441a89c2c78769ebcbbb2ba6645d7c1b117f67302caedff39dd5308310cddda"} Nov 01 16:08:16.717937 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:16.717845 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" podStartSLOduration=1.717832528 podStartE2EDuration="1.717832528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:16.71619341 +0000 UTC m=+6635.774345262" watchObservedRunningTime="2026-11-01 16:08:16.717832528 +0000 UTC m=+6635.775984387" Nov 01 16:08:17.611128 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:17.611092 5473 generic.go:356] "Generic (PLEG): container finished" podID="243fd851-1e2a-47a3-bd76-ba213767796a" containerID="8d1b7609c28c6561f530b829ac5288eec0f8219eaec32672d8bca7cfdfb06877" exitCode=0 Nov 01 16:08:17.611560 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:17.611180 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" event={"ID":"243fd851-1e2a-47a3-bd76-ba213767796a","Type":"ContainerDied","Data":"8d1b7609c28c6561f530b829ac5288eec0f8219eaec32672d8bca7cfdfb06877"} Nov 01 16:08:18.453072 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.453040 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:18.747185 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.747157 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:18.788376 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.788349 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc\" (UniqueName: \"kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc\") pod \"243fd851-1e2a-47a3-bd76-ba213767796a\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " Nov 01 16:08:18.788514 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.788411 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca\") pod \"243fd851-1e2a-47a3-bd76-ba213767796a\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " Nov 01 16:08:18.788514 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.788492 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca\") pod \"243fd851-1e2a-47a3-bd76-ba213767796a\" (UID: \"243fd851-1e2a-47a3-bd76-ba213767796a\") " Nov 01 16:08:18.788861 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.788832 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca" pod "243fd851-1e2a-47a3-bd76-ba213767796a" (UID: "243fd851-1e2a-47a3-bd76-ba213767796a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:18.790488 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.790470 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc" pod "243fd851-1e2a-47a3-bd76-ba213767796a" (UID: "243fd851-1e2a-47a3-bd76-ba213767796a"). InnerVolumeSpecName "kube-api-access-b6thc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:18.790552 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.790469 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca" pod "243fd851-1e2a-47a3-bd76-ba213767796a" (UID: "243fd851-1e2a-47a3-bd76-ba213767796a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:18.890261 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.890232 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:18.890261 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.890257 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6thc\" (UniqueName: \"kubernetes.io/projected/243fd851-1e2a-47a3-bd76-ba213767796a-kube-api-access-b6thc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:18.890261 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:18.890268 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243fd851-1e2a-47a3-bd76-ba213767796a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:19.046576 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:19.046538 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz"] Nov 01 16:08:19.048939 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:19.048916 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-pfzpz"] Nov 01 16:08:19.080984 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:19.080955 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27b5314-c52b-4a79-9746-d88ad733fce1" path="/var/lib/kubelet/pods/d27b5314-c52b-4a79-9746-d88ad733fce1/volumes" Nov 01 16:08:19.624042 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:19.624012 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" Nov 01 16:08:19.624240 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:19.624045 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql" event={"ID":"243fd851-1e2a-47a3-bd76-ba213767796a","Type":"ContainerDied","Data":"3441a89c2c78769ebcbbb2ba6645d7c1b117f67302caedff39dd5308310cddda"} Nov 01 16:08:19.624240 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:19.624097 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3441a89c2c78769ebcbbb2ba6645d7c1b117f67302caedff39dd5308310cddda" Nov 01 16:08:31.106054 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:08:31.106015 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:08:42.583238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.581310 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:08:42.583238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.581409 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:42.604236 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.603325 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:08:42.604236 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.603380 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:42.647336 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.647303 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:08:42.647618 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.647589 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:42.814350 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.814312 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:08:42.814670 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:42.814605 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:45.925294 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:45.925254 5473 scope.go:122] "RemoveContainer" containerID="d1837473d1dc73393abdfe079272fb14cac95a2084a205664004c07135b394ac" Nov 01 16:08:45.934615 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:45.934595 5473 scope.go:122] "RemoveContainer" containerID="88ed0d949e84b36c178700f02cb66548f5303949ca9518919175835f95b57f2f" Nov 01 16:08:45.944630 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:45.944602 5473 scope.go:122] "RemoveContainer" containerID="1e5cb9d0f6b2f1272435771686366ada5c7128464a940189659bc590fc402323" Nov 01 16:08:53.410978 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.410902 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp"] Nov 01 16:08:53.411475 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.411356 5473 cpu_manager.go:414] "containerMap: removing container" podUID="243fd851-1e2a-47a3-bd76-ba213767796a" containerName="revoke-keys" Nov 01 16:08:53.411475 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.411370 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243fd851-1e2a-47a3-bd76-ba213767796a" containerName="revoke-keys" Nov 01 16:08:53.411475 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.411432 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="243fd851-1e2a-47a3-bd76-ba213767796a" containerName="revoke-keys" Nov 01 16:08:53.416309 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.416292 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.469316 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.469289 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:08:53.477142 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.477112 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxmhv\" (UniqueName: \"kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.477347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.477162 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.477347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.477303 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.518953 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.518931 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:08:53.521195 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.521173 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp"] Nov 01 16:08:53.577814 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.577791 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxmhv\" (UniqueName: \"kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.577960 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.577825 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.577960 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.577860 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.578514 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.578484 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.580342 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.580317 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.637054 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.637033 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxmhv\" (UniqueName: \"kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv\") pod \"maas-api-revoke-sub-d855e84b-jmtnp\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:53.729122 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:53.729054 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:54.096961 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:54.096929 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp"] Nov 01 16:08:54.098798 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:08:54.098773 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060e2d94_0ea3_48cf_975d_17f0076df73a.slice/crio-4a70606b26acf2150c2f28dbfd3c5aa0d40708904cd3a0e199339dcbf00403b0 WatchSource:0}: Error finding container 4a70606b26acf2150c2f28dbfd3c5aa0d40708904cd3a0e199339dcbf00403b0: Status 404 returned error can't find the container with id 4a70606b26acf2150c2f28dbfd3c5aa0d40708904cd3a0e199339dcbf00403b0 Nov 01 16:08:54.889232 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:54.889171 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" event={"ID":"060e2d94-0ea3-48cf-975d-17f0076df73a","Type":"ContainerStarted","Data":"7f1644a2b25b14ce166c790118fc8ba39482b5d47779b7d7e205d711921f40b0"} Nov 01 16:08:54.889667 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:54.889247 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" event={"ID":"060e2d94-0ea3-48cf-975d-17f0076df73a","Type":"ContainerStarted","Data":"4a70606b26acf2150c2f28dbfd3c5aa0d40708904cd3a0e199339dcbf00403b0"} Nov 01 16:08:55.004094 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:55.004034 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" podStartSLOduration=2.004021034 podStartE2EDuration="2.004021034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:55.002422107 +0000 UTC m=+6674.060573969" watchObservedRunningTime="2026-11-01 16:08:55.004021034 +0000 UTC m=+6674.062172884" Nov 01 16:08:55.899192 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:55.899158 5473 generic.go:356] "Generic (PLEG): container finished" podID="060e2d94-0ea3-48cf-975d-17f0076df73a" containerID="7f1644a2b25b14ce166c790118fc8ba39482b5d47779b7d7e205d711921f40b0" exitCode=0 Nov 01 16:08:55.899704 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:55.899241 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" event={"ID":"060e2d94-0ea3-48cf-975d-17f0076df73a","Type":"ContainerDied","Data":"7f1644a2b25b14ce166c790118fc8ba39482b5d47779b7d7e205d711921f40b0"} Nov 01 16:08:57.034883 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.034853 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:57.109734 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.109704 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca\") pod \"060e2d94-0ea3-48cf-975d-17f0076df73a\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " Nov 01 16:08:57.109890 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.109741 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv\" (UniqueName: \"kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv\") pod \"060e2d94-0ea3-48cf-975d-17f0076df73a\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " Nov 01 16:08:57.109890 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.109839 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca\") pod \"060e2d94-0ea3-48cf-975d-17f0076df73a\" (UID: \"060e2d94-0ea3-48cf-975d-17f0076df73a\") " Nov 01 16:08:57.110071 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.110042 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca" pod "060e2d94-0ea3-48cf-975d-17f0076df73a" (UID: "060e2d94-0ea3-48cf-975d-17f0076df73a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:57.110139 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.110124 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.111957 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.111932 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv" pod "060e2d94-0ea3-48cf-975d-17f0076df73a" (UID: "060e2d94-0ea3-48cf-975d-17f0076df73a"). InnerVolumeSpecName "kube-api-access-mxmhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:57.112070 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.112011 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca" pod "060e2d94-0ea3-48cf-975d-17f0076df73a" (UID: "060e2d94-0ea3-48cf-975d-17f0076df73a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:57.210699 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.210624 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060e2d94-0ea3-48cf-975d-17f0076df73a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.210699 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.210652 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxmhv\" (UniqueName: \"kubernetes.io/projected/060e2d94-0ea3-48cf-975d-17f0076df73a-kube-api-access-mxmhv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.912161 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.912127 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" Nov 01 16:08:57.912343 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.912155 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp" event={"ID":"060e2d94-0ea3-48cf-975d-17f0076df73a","Type":"ContainerDied","Data":"4a70606b26acf2150c2f28dbfd3c5aa0d40708904cd3a0e199339dcbf00403b0"} Nov 01 16:08:57.912343 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:08:57.912240 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a70606b26acf2150c2f28dbfd3c5aa0d40708904cd3a0e199339dcbf00403b0" Nov 01 16:09:01.106820 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:09:01.106785 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:09:02.486545 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:02.486511 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:03.677029 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:03.676991 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:04.118101 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.118066 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp"] Nov 01 16:09:04.118535 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.118520 5473 cpu_manager.go:414] "containerMap: removing container" podUID="060e2d94-0ea3-48cf-975d-17f0076df73a" containerName="revoke-keys" Nov 01 16:09:04.118594 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.118537 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060e2d94-0ea3-48cf-975d-17f0076df73a" containerName="revoke-keys" Nov 01 16:09:04.118634 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.118613 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="060e2d94-0ea3-48cf-975d-17f0076df73a" containerName="revoke-keys" Nov 01 16:09:04.121900 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.121877 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.174762 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.174725 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:09:04.177973 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.177945 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp"] Nov 01 16:09:04.182264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.182236 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.182429 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.182410 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjd8\" (UniqueName: \"kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.182502 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.182459 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.224610 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.224588 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:09:04.283264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.283235 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjd8\" (UniqueName: \"kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.283405 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.283283 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.283405 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.283319 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.283961 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.283930 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.285631 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.285610 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.339687 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.339656 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjd8\" (UniqueName: \"kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8\") pod \"maas-api-revoke-sub-e98ff841-7v9fp\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.435429 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.435360 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:04.805722 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.805692 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp"] Nov 01 16:09:04.807670 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:09:04.807637 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65a5088_9995_43de_9295_62f1d36cf1b9.slice/crio-957d028b218ca8d6082fc30fcaea252c9585278a9e74a640b972c6f5fe75b9ce WatchSource:0}: Error finding container 957d028b218ca8d6082fc30fcaea252c9585278a9e74a640b972c6f5fe75b9ce: Status 404 returned error can't find the container with id 957d028b218ca8d6082fc30fcaea252c9585278a9e74a640b972c6f5fe75b9ce Nov 01 16:09:04.952366 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:04.952326 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" event={"ID":"d65a5088-9995-43de-9295-62f1d36cf1b9","Type":"ContainerStarted","Data":"957d028b218ca8d6082fc30fcaea252c9585278a9e74a640b972c6f5fe75b9ce"} Nov 01 16:09:05.959193 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:05.959106 5473 generic.go:356] "Generic (PLEG): container finished" podID="d65a5088-9995-43de-9295-62f1d36cf1b9" containerID="bb1d77b1211e48720c87db0bfc9da0925c765ed8885652d471a217a98c3c8ca3" exitCode=0 Nov 01 16:09:05.959597 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:05.959195 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" event={"ID":"d65a5088-9995-43de-9295-62f1d36cf1b9","Type":"ContainerDied","Data":"bb1d77b1211e48720c87db0bfc9da0925c765ed8885652d471a217a98c3c8ca3"} Nov 01 16:09:07.099587 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.099562 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:07.210705 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.210674 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca\") pod \"d65a5088-9995-43de-9295-62f1d36cf1b9\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " Nov 01 16:09:07.210866 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.210783 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8\" (UniqueName: \"kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8\") pod \"d65a5088-9995-43de-9295-62f1d36cf1b9\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " Nov 01 16:09:07.210956 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.210937 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca\") pod \"d65a5088-9995-43de-9295-62f1d36cf1b9\" (UID: \"d65a5088-9995-43de-9295-62f1d36cf1b9\") " Nov 01 16:09:07.211120 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.211097 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca" pod "d65a5088-9995-43de-9295-62f1d36cf1b9" (UID: "d65a5088-9995-43de-9295-62f1d36cf1b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:07.211480 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.211456 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:07.214182 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.214151 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca" pod "d65a5088-9995-43de-9295-62f1d36cf1b9" (UID: "d65a5088-9995-43de-9295-62f1d36cf1b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:07.215293 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.215211 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8" pod "d65a5088-9995-43de-9295-62f1d36cf1b9" (UID: "d65a5088-9995-43de-9295-62f1d36cf1b9"). InnerVolumeSpecName "kube-api-access-mqjd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:07.312605 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.312565 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjd8\" (UniqueName: \"kubernetes.io/projected/d65a5088-9995-43de-9295-62f1d36cf1b9-kube-api-access-mqjd8\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:07.312605 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.312598 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65a5088-9995-43de-9295-62f1d36cf1b9-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:07.971137 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.971104 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" event={"ID":"d65a5088-9995-43de-9295-62f1d36cf1b9","Type":"ContainerDied","Data":"957d028b218ca8d6082fc30fcaea252c9585278a9e74a640b972c6f5fe75b9ce"} Nov 01 16:09:07.971357 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.971159 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957d028b218ca8d6082fc30fcaea252c9585278a9e74a640b972c6f5fe75b9ce" Nov 01 16:09:07.971357 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:07.971119 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp" Nov 01 16:09:18.710558 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:18.710521 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.109705 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:09:31.109663 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:09:37.258672 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.258628 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl"] Nov 01 16:09:37.259354 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.259327 5473 cpu_manager.go:414] "containerMap: removing container" podUID="d65a5088-9995-43de-9295-62f1d36cf1b9" containerName="revoke-keys" Nov 01 16:09:37.259490 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.259356 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65a5088-9995-43de-9295-62f1d36cf1b9" containerName="revoke-keys" Nov 01 16:09:37.259548 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.259537 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65a5088-9995-43de-9295-62f1d36cf1b9" containerName="revoke-keys" Nov 01 16:09:37.263200 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.263177 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.319858 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.319810 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl"] Nov 01 16:09:37.341363 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.341331 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.341567 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.341368 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.341567 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.341399 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fnv\" (UniqueName: \"kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.365525 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.365495 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:09:37.365720 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.365536 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:09:37.443120 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.443082 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.443365 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.443127 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.443365 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.443166 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fnv\" (UniqueName: \"kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.443918 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.443894 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.445622 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.445599 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.499872 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.499840 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fnv\" (UniqueName: \"kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv\") pod \"maas-api-revoke-sub-fc9279f4-4lnbl\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.580135 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.580105 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:37.949745 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:37.949720 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl"] Nov 01 16:09:37.950535 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:09:37.950509 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef9c557_b2aa_48b8_9700_04a89aca9b31.slice/crio-b67f910af83506254b8ca8d090cd4aad328b16436ad41973fa6a880e3aee9d2e WatchSource:0}: Error finding container b67f910af83506254b8ca8d090cd4aad328b16436ad41973fa6a880e3aee9d2e: Status 404 returned error can't find the container with id b67f910af83506254b8ca8d090cd4aad328b16436ad41973fa6a880e3aee9d2e Nov 01 16:09:38.179606 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:38.179557 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" event={"ID":"8ef9c557-b2aa-48b8-9700-04a89aca9b31","Type":"ContainerStarted","Data":"b67f910af83506254b8ca8d090cd4aad328b16436ad41973fa6a880e3aee9d2e"} Nov 01 16:09:39.186256 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:39.186160 5473 generic.go:356] "Generic (PLEG): container finished" podID="8ef9c557-b2aa-48b8-9700-04a89aca9b31" containerID="71cf9004be1aa112c3b3236707b7621ea6752c5817b573453f3f2b3a10e99df6" exitCode=0 Nov 01 16:09:39.186256 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:39.186203 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" event={"ID":"8ef9c557-b2aa-48b8-9700-04a89aca9b31","Type":"ContainerDied","Data":"71cf9004be1aa112c3b3236707b7621ea6752c5817b573453f3f2b3a10e99df6"} Nov 01 16:09:40.321998 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.321973 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:40.373119 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.373094 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca\") pod \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " Nov 01 16:09:40.373258 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.373168 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca\") pod \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " Nov 01 16:09:40.373258 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.373198 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv\" (UniqueName: \"kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv\") pod \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\" (UID: \"8ef9c557-b2aa-48b8-9700-04a89aca9b31\") " Nov 01 16:09:40.373470 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.373446 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca" pod "8ef9c557-b2aa-48b8-9700-04a89aca9b31" (UID: "8ef9c557-b2aa-48b8-9700-04a89aca9b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:40.373560 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.373524 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:40.375279 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.375251 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca" pod "8ef9c557-b2aa-48b8-9700-04a89aca9b31" (UID: "8ef9c557-b2aa-48b8-9700-04a89aca9b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:40.375376 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.375304 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv" pod "8ef9c557-b2aa-48b8-9700-04a89aca9b31" (UID: "8ef9c557-b2aa-48b8-9700-04a89aca9b31"). InnerVolumeSpecName "kube-api-access-k5fnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:40.474258 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.474172 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef9c557-b2aa-48b8-9700-04a89aca9b31-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:40.474258 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:40.474201 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5fnv\" (UniqueName: \"kubernetes.io/projected/8ef9c557-b2aa-48b8-9700-04a89aca9b31-kube-api-access-k5fnv\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:41.198006 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:41.197976 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:09:41.198206 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:41.198017 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" event={"ID":"8ef9c557-b2aa-48b8-9700-04a89aca9b31","Type":"ContainerDied","Data":"b67f910af83506254b8ca8d090cd4aad328b16436ad41973fa6a880e3aee9d2e"} Nov 01 16:09:41.198206 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:41.198061 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67f910af83506254b8ca8d090cd4aad328b16436ad41973fa6a880e3aee9d2e" Nov 01 16:09:45.972977 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:45.972938 5473 scope.go:122] "RemoveContainer" containerID="4e3e540e6a0ab9cf5873b2d57dc414aadfc2a7a8e5708978154e41ff74c8ecbe" Nov 01 16:09:45.982379 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:45.982349 5473 scope.go:122] "RemoveContainer" containerID="831d2e538b1a9151998c1cfe29ad3d73c17c0515da995a32f8ea6b1441443d1e" Nov 01 16:09:45.990477 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:09:45.990459 5473 scope.go:122] "RemoveContainer" containerID="658cd051e93fd479e7ce12d167e8f2703311880da6fc51b6b86219c02d32cb25" Nov 01 16:10:01.106570 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:10:01.106532 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:10:08.048451 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:08.048413 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr"] Nov 01 16:10:08.098049 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:08.098020 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93ac944-dcptr"] Nov 01 16:10:09.080126 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:09.080091 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05024251-322e-4d9e-867a-075108dc785a" path="/var/lib/kubelet/pods/05024251-322e-4d9e-867a-075108dc785a/volumes" Nov 01 16:10:19.412096 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.412059 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb"] Nov 01 16:10:19.412519 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.412486 5473 cpu_manager.go:414] "containerMap: removing container" podUID="8ef9c557-b2aa-48b8-9700-04a89aca9b31" containerName="revoke-keys" Nov 01 16:10:19.412519 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.412499 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef9c557-b2aa-48b8-9700-04a89aca9b31" containerName="revoke-keys" Nov 01 16:10:19.412603 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.412567 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef9c557-b2aa-48b8-9700-04a89aca9b31" containerName="revoke-keys" Nov 01 16:10:19.415983 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.415965 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.418594 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.418569 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.418720 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.418680 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.418785 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.418730 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89skk\" (UniqueName: \"kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.518883 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.518854 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:10:19.519076 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.518959 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:10:19.519412 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.519391 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.519469 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.519441 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89skk\" (UniqueName: \"kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.519524 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.519485 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.520201 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.520183 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.521652 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.521632 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb"] Nov 01 16:10:19.521906 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.521889 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.575692 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.575665 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89skk\" (UniqueName: \"kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk\") pod \"maas-api-revoke-sub-f9ff808b-t6ggb\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:19.728604 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:19.728510 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:20.101488 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:20.101459 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb"] Nov 01 16:10:20.102980 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:10:20.102954 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c42abd1_977c_48b3_92d9_cab1b134de8b.slice/crio-4fe00804dfce6be6a8ee67a1c6d8dedfa32f6d2876b311c2ba84ab2f8f00da5c WatchSource:0}: Error finding container 4fe00804dfce6be6a8ee67a1c6d8dedfa32f6d2876b311c2ba84ab2f8f00da5c: Status 404 returned error can't find the container with id 4fe00804dfce6be6a8ee67a1c6d8dedfa32f6d2876b311c2ba84ab2f8f00da5c Nov 01 16:10:20.496802 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:20.496760 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" event={"ID":"9c42abd1-977c-48b3-92d9-cab1b134de8b","Type":"ContainerStarted","Data":"9722662ea74dff1f4e101c8f2efa3ffe2bdfa4e921b755b05771cbe047ac14e8"} Nov 01 16:10:20.497277 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:20.496823 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" event={"ID":"9c42abd1-977c-48b3-92d9-cab1b134de8b","Type":"ContainerStarted","Data":"4fe00804dfce6be6a8ee67a1c6d8dedfa32f6d2876b311c2ba84ab2f8f00da5c"} Nov 01 16:10:20.612725 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:20.612673 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" podStartSLOduration=1.6126599069999998 podStartE2EDuration="1.612659907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:20.60999465 +0000 UTC m=+6759.668146503" watchObservedRunningTime="2026-11-01 16:10:20.612659907 +0000 UTC m=+6759.670811757" Nov 01 16:10:20.786766 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:20.786733 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:21.503770 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:21.503741 5473 generic.go:356] "Generic (PLEG): container finished" podID="9c42abd1-977c-48b3-92d9-cab1b134de8b" containerID="9722662ea74dff1f4e101c8f2efa3ffe2bdfa4e921b755b05771cbe047ac14e8" exitCode=0 Nov 01 16:10:21.504241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:21.503786 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" event={"ID":"9c42abd1-977c-48b3-92d9-cab1b134de8b","Type":"ContainerDied","Data":"9722662ea74dff1f4e101c8f2efa3ffe2bdfa4e921b755b05771cbe047ac14e8"} Nov 01 16:10:22.640909 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.640879 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:22.749891 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.749861 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca\") pod \"9c42abd1-977c-48b3-92d9-cab1b134de8b\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " Nov 01 16:10:22.750054 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.749911 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca\") pod \"9c42abd1-977c-48b3-92d9-cab1b134de8b\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " Nov 01 16:10:22.750054 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.750021 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk\" (UniqueName: \"kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk\") pod \"9c42abd1-977c-48b3-92d9-cab1b134de8b\" (UID: \"9c42abd1-977c-48b3-92d9-cab1b134de8b\") " Nov 01 16:10:22.750262 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.750238 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca" pod "9c42abd1-977c-48b3-92d9-cab1b134de8b" (UID: "9c42abd1-977c-48b3-92d9-cab1b134de8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:22.750390 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.750373 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:22.752058 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.752038 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca" pod "9c42abd1-977c-48b3-92d9-cab1b134de8b" (UID: "9c42abd1-977c-48b3-92d9-cab1b134de8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:22.752163 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.752099 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk" pod "9c42abd1-977c-48b3-92d9-cab1b134de8b" (UID: "9c42abd1-977c-48b3-92d9-cab1b134de8b"). InnerVolumeSpecName "kube-api-access-89skk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:22.851204 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.851171 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c42abd1-977c-48b3-92d9-cab1b134de8b-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:22.851204 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:22.851199 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89skk\" (UniqueName: \"kubernetes.io/projected/9c42abd1-977c-48b3-92d9-cab1b134de8b-kube-api-access-89skk\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:23.516122 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:23.516089 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" Nov 01 16:10:23.516317 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:23.516095 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb" event={"ID":"9c42abd1-977c-48b3-92d9-cab1b134de8b","Type":"ContainerDied","Data":"4fe00804dfce6be6a8ee67a1c6d8dedfa32f6d2876b311c2ba84ab2f8f00da5c"} Nov 01 16:10:23.516317 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:23.516242 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fe00804dfce6be6a8ee67a1c6d8dedfa32f6d2876b311c2ba84ab2f8f00da5c" Nov 01 16:10:30.126750 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.126714 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c"] Nov 01 16:10:30.127320 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.127301 5473 cpu_manager.go:414] "containerMap: removing container" podUID="9c42abd1-977c-48b3-92d9-cab1b134de8b" containerName="revoke-keys" Nov 01 16:10:30.127433 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.127322 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c42abd1-977c-48b3-92d9-cab1b134de8b" containerName="revoke-keys" Nov 01 16:10:30.127495 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.127442 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c42abd1-977c-48b3-92d9-cab1b134de8b" containerName="revoke-keys" Nov 01 16:10:30.130813 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.130792 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.188697 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.188654 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c"] Nov 01 16:10:30.220075 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.220043 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh64\" (UniqueName: \"kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.220277 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.220085 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.220277 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.220238 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.235265 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.235246 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:10:30.235370 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.235354 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:10:30.321188 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.321160 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.321347 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.321313 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh64\" (UniqueName: \"kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.321402 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.321349 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.321981 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.321963 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.323541 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.323521 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.377788 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.377725 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh64\" (UniqueName: \"kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64\") pod \"maas-api-revoke-sub-02ce465c-tv82c\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.443967 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.443942 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:30.811420 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:30.811390 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c"] Nov 01 16:10:30.812956 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:10:30.812927 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270fd8ca_10aa_4876_a14b_8772dc422587.slice/crio-f4f9cf1b60c2e0d8b1d3a7e14465cf7d51e4609666c998fdacb97523a6e4b39a WatchSource:0}: Error finding container f4f9cf1b60c2e0d8b1d3a7e14465cf7d51e4609666c998fdacb97523a6e4b39a: Status 404 returned error can't find the container with id f4f9cf1b60c2e0d8b1d3a7e14465cf7d51e4609666c998fdacb97523a6e4b39a Nov 01 16:10:31.125919 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:10:31.125883 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:10:31.563156 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:31.563117 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" event={"ID":"270fd8ca-10aa-4876-a14b-8772dc422587","Type":"ContainerStarted","Data":"e9fdaba1d9d67a34fe0ad1271a6db64c60709548d7f4cd75d7b51592c9c0e517"} Nov 01 16:10:31.563565 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:31.563180 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" event={"ID":"270fd8ca-10aa-4876-a14b-8772dc422587","Type":"ContainerStarted","Data":"f4f9cf1b60c2e0d8b1d3a7e14465cf7d51e4609666c998fdacb97523a6e4b39a"} Nov 01 16:10:31.678611 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:31.678560 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" podStartSLOduration=1.6785446149999999 podStartE2EDuration="1.678544615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:31.675690028 +0000 UTC m=+6770.733841882" watchObservedRunningTime="2026-11-01 16:10:31.678544615 +0000 UTC m=+6770.736696464" Nov 01 16:10:32.569828 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:32.569792 5473 generic.go:356] "Generic (PLEG): container finished" podID="270fd8ca-10aa-4876-a14b-8772dc422587" containerID="e9fdaba1d9d67a34fe0ad1271a6db64c60709548d7f4cd75d7b51592c9c0e517" exitCode=0 Nov 01 16:10:32.570502 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:32.569860 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" event={"ID":"270fd8ca-10aa-4876-a14b-8772dc422587","Type":"ContainerDied","Data":"e9fdaba1d9d67a34fe0ad1271a6db64c60709548d7f4cd75d7b51592c9c0e517"} Nov 01 16:10:33.709918 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.709893 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:33.754113 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.754086 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64\" (UniqueName: \"kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64\") pod \"270fd8ca-10aa-4876-a14b-8772dc422587\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " Nov 01 16:10:33.754274 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.754169 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca\") pod \"270fd8ca-10aa-4876-a14b-8772dc422587\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " Nov 01 16:10:33.754274 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.754198 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca\") pod \"270fd8ca-10aa-4876-a14b-8772dc422587\" (UID: \"270fd8ca-10aa-4876-a14b-8772dc422587\") " Nov 01 16:10:33.754517 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.754492 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca" pod "270fd8ca-10aa-4876-a14b-8772dc422587" (UID: "270fd8ca-10aa-4876-a14b-8772dc422587"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:33.754638 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.754594 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:33.756196 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.756172 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64" pod "270fd8ca-10aa-4876-a14b-8772dc422587" (UID: "270fd8ca-10aa-4876-a14b-8772dc422587"). InnerVolumeSpecName "kube-api-access-mkh64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:33.756290 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.756267 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca" pod "270fd8ca-10aa-4876-a14b-8772dc422587" (UID: "270fd8ca-10aa-4876-a14b-8772dc422587"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:33.855384 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.855313 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/270fd8ca-10aa-4876-a14b-8772dc422587-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:33.855384 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:33.855345 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkh64\" (UniqueName: \"kubernetes.io/projected/270fd8ca-10aa-4876-a14b-8772dc422587-kube-api-access-mkh64\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:34.581307 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:34.581274 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" Nov 01 16:10:34.581489 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:34.581281 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c" event={"ID":"270fd8ca-10aa-4876-a14b-8772dc422587","Type":"ContainerDied","Data":"f4f9cf1b60c2e0d8b1d3a7e14465cf7d51e4609666c998fdacb97523a6e4b39a"} Nov 01 16:10:34.581489 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:34.581401 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4f9cf1b60c2e0d8b1d3a7e14465cf7d51e4609666c998fdacb97523a6e4b39a" Nov 01 16:10:53.196993 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:53.196959 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:57.582241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:57.578344 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:10:57.582241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:57.578424 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:10:57.582241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:57.578882 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:10:57.582241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:10:57.578924 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.107198 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:11:01.107161 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:11:06.048560 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:06.048507 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw"] Nov 01 16:11:06.105601 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:06.105566 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-btgcw"] Nov 01 16:11:07.108241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:07.104741 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bc5427-8176-480d-9dbd-d6f9b26a2079" path="/var/lib/kubelet/pods/03bc5427-8176-480d-9dbd-d6f9b26a2079/volumes" Nov 01 16:11:15.482171 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.482136 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh"] Nov 01 16:11:15.482601 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.482564 5473 cpu_manager.go:414] "containerMap: removing container" podUID="270fd8ca-10aa-4876-a14b-8772dc422587" containerName="revoke-keys" Nov 01 16:11:15.482601 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.482576 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270fd8ca-10aa-4876-a14b-8772dc422587" containerName="revoke-keys" Nov 01 16:11:15.482696 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.482658 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="270fd8ca-10aa-4876-a14b-8772dc422587" containerName="revoke-keys" Nov 01 16:11:15.486039 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.486019 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.539416 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.539365 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:11:15.539803 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.539778 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:11:15.542154 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.542128 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh"] Nov 01 16:11:15.593393 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.593351 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.593393 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.593398 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvphh\" (UniqueName: \"kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.593661 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.593627 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.695152 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.695119 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.695348 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.695171 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.695348 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.695282 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvphh\" (UniqueName: \"kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.695793 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.695767 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.697635 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.697613 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.752374 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.752315 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvphh\" (UniqueName: \"kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh\") pod \"maas-api-revoke-sub-53509b25-z64nh\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:15.799361 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:15.799338 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:16.168948 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:16.168922 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh"] Nov 01 16:11:16.171807 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:11:16.171777 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730900f8_2e88_46be_b3d1_a6cc250db6a8.slice/crio-509655f7474a9d25282c510aa55e6f4102e4f27b25e86343904d16965dad0836 WatchSource:0}: Error finding container 509655f7474a9d25282c510aa55e6f4102e4f27b25e86343904d16965dad0836: Status 404 returned error can't find the container with id 509655f7474a9d25282c510aa55e6f4102e4f27b25e86343904d16965dad0836 Nov 01 16:11:16.930596 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:16.930556 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" event={"ID":"730900f8-2e88-46be-b3d1-a6cc250db6a8","Type":"ContainerStarted","Data":"4b8ebce73bb7cc3b728addc35977f2b04349d33c3012b634ada73b9e73f890e0"} Nov 01 16:11:16.931057 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:16.930615 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" event={"ID":"730900f8-2e88-46be-b3d1-a6cc250db6a8","Type":"ContainerStarted","Data":"509655f7474a9d25282c510aa55e6f4102e4f27b25e86343904d16965dad0836"} Nov 01 16:11:17.046611 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:17.046550 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" podStartSLOduration=2.046535805 podStartE2EDuration="2.046535805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:17.043841204 +0000 UTC m=+6816.101993037" watchObservedRunningTime="2026-11-01 16:11:17.046535805 +0000 UTC m=+6816.104687686" Nov 01 16:11:17.938877 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:17.938840 5473 generic.go:356] "Generic (PLEG): container finished" podID="730900f8-2e88-46be-b3d1-a6cc250db6a8" containerID="4b8ebce73bb7cc3b728addc35977f2b04349d33c3012b634ada73b9e73f890e0" exitCode=0 Nov 01 16:11:17.939337 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:17.938911 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" event={"ID":"730900f8-2e88-46be-b3d1-a6cc250db6a8","Type":"ContainerDied","Data":"4b8ebce73bb7cc3b728addc35977f2b04349d33c3012b634ada73b9e73f890e0"} Nov 01 16:11:19.071140 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.071116 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:19.130770 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.130734 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca\") pod \"730900f8-2e88-46be-b3d1-a6cc250db6a8\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " Nov 01 16:11:19.130918 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.130782 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh\" (UniqueName: \"kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh\") pod \"730900f8-2e88-46be-b3d1-a6cc250db6a8\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " Nov 01 16:11:19.130918 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.130851 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca\") pod \"730900f8-2e88-46be-b3d1-a6cc250db6a8\" (UID: \"730900f8-2e88-46be-b3d1-a6cc250db6a8\") " Nov 01 16:11:19.131275 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.131251 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca" pod "730900f8-2e88-46be-b3d1-a6cc250db6a8" (UID: "730900f8-2e88-46be-b3d1-a6cc250db6a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:19.132842 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.132810 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca" pod "730900f8-2e88-46be-b3d1-a6cc250db6a8" (UID: "730900f8-2e88-46be-b3d1-a6cc250db6a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:19.132953 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.132934 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh" pod "730900f8-2e88-46be-b3d1-a6cc250db6a8" (UID: "730900f8-2e88-46be-b3d1-a6cc250db6a8"). InnerVolumeSpecName "kube-api-access-cvphh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:19.231979 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.231901 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvphh\" (UniqueName: \"kubernetes.io/projected/730900f8-2e88-46be-b3d1-a6cc250db6a8-kube-api-access-cvphh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:19.231979 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.231927 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:19.231979 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.231940 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/730900f8-2e88-46be-b3d1-a6cc250db6a8-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:19.353102 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.353058 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:19.951905 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.951876 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" Nov 01 16:11:19.952098 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.951909 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53509b25-z64nh" event={"ID":"730900f8-2e88-46be-b3d1-a6cc250db6a8","Type":"ContainerDied","Data":"509655f7474a9d25282c510aa55e6f4102e4f27b25e86343904d16965dad0836"} Nov 01 16:11:19.952098 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:19.951955 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="509655f7474a9d25282c510aa55e6f4102e4f27b25e86343904d16965dad0836" Nov 01 16:11:26.282063 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.282026 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h"] Nov 01 16:11:26.282642 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.282622 5473 cpu_manager.go:414] "containerMap: removing container" podUID="730900f8-2e88-46be-b3d1-a6cc250db6a8" containerName="revoke-keys" Nov 01 16:11:26.282732 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.282645 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730900f8-2e88-46be-b3d1-a6cc250db6a8" containerName="revoke-keys" Nov 01 16:11:26.282794 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.282784 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="730900f8-2e88-46be-b3d1-a6cc250db6a8" containerName="revoke-keys" Nov 01 16:11:26.286160 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.286135 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.340462 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.340433 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:11:26.340671 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.340628 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:11:26.344264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.344240 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h"] Nov 01 16:11:26.402621 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.402590 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfpdw\" (UniqueName: \"kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.402802 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.402721 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.402848 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.402805 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.504054 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.504025 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfpdw\" (UniqueName: \"kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.504273 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.504104 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.504273 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.504154 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.504755 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.504735 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.506330 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.506306 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.560874 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.560809 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfpdw\" (UniqueName: \"kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw\") pod \"maas-api-revoke-sub-3bdbc843-7rx8h\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.598799 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.598774 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:26.970386 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.970343 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h"] Nov 01 16:11:26.971697 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:11:26.971671 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98743823_17b3_41b7_8984_7bbfaf40f94a.slice/crio-242e3ef7ad8ccefb89282e8c49ef5b04a6a1b4885fe0bfcbd6d5f78cd317929a WatchSource:0}: Error finding container 242e3ef7ad8ccefb89282e8c49ef5b04a6a1b4885fe0bfcbd6d5f78cd317929a: Status 404 returned error can't find the container with id 242e3ef7ad8ccefb89282e8c49ef5b04a6a1b4885fe0bfcbd6d5f78cd317929a Nov 01 16:11:26.992205 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:26.992182 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" event={"ID":"98743823-17b3-41b7-8984-7bbfaf40f94a","Type":"ContainerStarted","Data":"242e3ef7ad8ccefb89282e8c49ef5b04a6a1b4885fe0bfcbd6d5f78cd317929a"} Nov 01 16:11:27.999808 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:27.999711 5473 generic.go:356] "Generic (PLEG): container finished" podID="98743823-17b3-41b7-8984-7bbfaf40f94a" containerID="bccdceede38a096b5e64028d826a17f6b04f242bc5a33606d3cf59c0fb812b72" exitCode=0 Nov 01 16:11:28.000206 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:27.999795 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" event={"ID":"98743823-17b3-41b7-8984-7bbfaf40f94a","Type":"ContainerDied","Data":"bccdceede38a096b5e64028d826a17f6b04f242bc5a33606d3cf59c0fb812b72"} Nov 01 16:11:29.212748 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.212716 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:29.331385 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.331286 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca\") pod \"98743823-17b3-41b7-8984-7bbfaf40f94a\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " Nov 01 16:11:29.331385 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.331362 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw\" (UniqueName: \"kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw\") pod \"98743823-17b3-41b7-8984-7bbfaf40f94a\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " Nov 01 16:11:29.331636 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.331403 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca\") pod \"98743823-17b3-41b7-8984-7bbfaf40f94a\" (UID: \"98743823-17b3-41b7-8984-7bbfaf40f94a\") " Nov 01 16:11:29.332185 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.332153 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca" pod "98743823-17b3-41b7-8984-7bbfaf40f94a" (UID: "98743823-17b3-41b7-8984-7bbfaf40f94a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:29.335149 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.334914 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca" pod "98743823-17b3-41b7-8984-7bbfaf40f94a" (UID: "98743823-17b3-41b7-8984-7bbfaf40f94a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:29.335149 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.335080 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw" pod "98743823-17b3-41b7-8984-7bbfaf40f94a" (UID: "98743823-17b3-41b7-8984-7bbfaf40f94a"). InnerVolumeSpecName "kube-api-access-nfpdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:29.433506 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.433458 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:29.433506 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.433502 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfpdw\" (UniqueName: \"kubernetes.io/projected/98743823-17b3-41b7-8984-7bbfaf40f94a-kube-api-access-nfpdw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:29.433804 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:29.433518 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98743823-17b3-41b7-8984-7bbfaf40f94a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:30.050239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:30.040956 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" event={"ID":"98743823-17b3-41b7-8984-7bbfaf40f94a","Type":"ContainerDied","Data":"242e3ef7ad8ccefb89282e8c49ef5b04a6a1b4885fe0bfcbd6d5f78cd317929a"} Nov 01 16:11:30.050239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:30.041025 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="242e3ef7ad8ccefb89282e8c49ef5b04a6a1b4885fe0bfcbd6d5f78cd317929a" Nov 01 16:11:30.050239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:30.041096 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bdbc843-7rx8h" Nov 01 16:11:31.249239 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:11:31.248610 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:11:46.038240 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:11:46.036153 5473 scope.go:122] "RemoveContainer" containerID="3cfb592f0bba2c01c4de87523756dcb9a42141848a1bf798e58a3369340a9695" Nov 01 16:12:01.107164 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:12:01.107082 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:12:13.880669 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.880632 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr"] Nov 01 16:12:13.881086 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.881073 5473 cpu_manager.go:414] "containerMap: removing container" podUID="98743823-17b3-41b7-8984-7bbfaf40f94a" containerName="revoke-keys" Nov 01 16:12:13.881134 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.881087 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98743823-17b3-41b7-8984-7bbfaf40f94a" containerName="revoke-keys" Nov 01 16:12:13.881176 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.881149 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="98743823-17b3-41b7-8984-7bbfaf40f94a" containerName="revoke-keys" Nov 01 16:12:13.884497 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.884480 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:13.909084 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.909046 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5654r\" (UniqueName: \"kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:13.909308 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.909091 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:13.909308 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.909246 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:13.941584 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.941548 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr"] Nov 01 16:12:13.987253 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.987205 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:13.987441 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:13.987370 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:12:14.010141 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.010107 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5654r\" (UniqueName: \"kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.010337 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.010148 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.010337 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.010208 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.010885 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.010860 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.012564 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.012539 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.067113 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.067078 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5654r\" (UniqueName: \"kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r\") pod \"maas-api-revoke-sub-3576bff8-92nvr\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.198056 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.197974 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:14.572084 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:14.572047 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr"] Nov 01 16:12:14.573609 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:12:14.573578 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0ee04a_fd7b_4c62_9f24_eb41bb05faad.slice/crio-5f860d2f45ec9a4191bce33420af1db76f8d35b3b1c4303dfc6929edd40f094e WatchSource:0}: Error finding container 5f860d2f45ec9a4191bce33420af1db76f8d35b3b1c4303dfc6929edd40f094e: Status 404 returned error can't find the container with id 5f860d2f45ec9a4191bce33420af1db76f8d35b3b1c4303dfc6929edd40f094e Nov 01 16:12:15.431699 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:15.431663 5473 generic.go:356] "Generic (PLEG): container finished" podID="1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" containerID="d6a86d02a2e9452d9f401e9aa9fc880e16038290d65bb94f25f708bd24710a01" exitCode=0 Nov 01 16:12:15.432146 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:15.431745 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" event={"ID":"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad","Type":"ContainerDied","Data":"d6a86d02a2e9452d9f401e9aa9fc880e16038290d65bb94f25f708bd24710a01"} Nov 01 16:12:15.432146 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:15.431811 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" event={"ID":"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad","Type":"ContainerStarted","Data":"5f860d2f45ec9a4191bce33420af1db76f8d35b3b1c4303dfc6929edd40f094e"} Nov 01 16:12:16.572668 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.572643 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:16.636121 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.636086 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r\" (UniqueName: \"kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r\") pod \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " Nov 01 16:12:16.636331 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.636203 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca\") pod \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " Nov 01 16:12:16.636331 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.636262 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca\") pod \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\" (UID: \"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad\") " Nov 01 16:12:16.636671 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.636637 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca" pod "1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" (UID: "1c0ee04a-fd7b-4c62-9f24-eb41bb05faad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:16.638395 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.638365 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca" pod "1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" (UID: "1c0ee04a-fd7b-4c62-9f24-eb41bb05faad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:16.638489 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.638400 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r" pod "1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" (UID: "1c0ee04a-fd7b-4c62-9f24-eb41bb05faad"). InnerVolumeSpecName "kube-api-access-5654r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:16.737591 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.737510 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:16.737591 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.737537 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:16.737591 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:16.737546 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5654r\" (UniqueName: \"kubernetes.io/projected/1c0ee04a-fd7b-4c62-9f24-eb41bb05faad-kube-api-access-5654r\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:17.444951 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:17.444859 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" Nov 01 16:12:17.445209 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:17.444860 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3576bff8-92nvr" event={"ID":"1c0ee04a-fd7b-4c62-9f24-eb41bb05faad","Type":"ContainerDied","Data":"5f860d2f45ec9a4191bce33420af1db76f8d35b3b1c4303dfc6929edd40f094e"} Nov 01 16:12:17.445209 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:17.444989 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f860d2f45ec9a4191bce33420af1db76f8d35b3b1c4303dfc6929edd40f094e" Nov 01 16:12:19.691027 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:19.690993 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:24.576050 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.576011 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl"] Nov 01 16:12:24.576467 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.576450 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" containerName="revoke-keys" Nov 01 16:12:24.576467 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.576463 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" containerName="revoke-keys" Nov 01 16:12:24.576562 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.576552 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0ee04a-fd7b-4c62-9f24-eb41bb05faad" containerName="revoke-keys" Nov 01 16:12:24.579910 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.579886 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.619170 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.619126 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brxsz\" (UniqueName: \"kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.619365 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.619195 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.619365 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.619262 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.635782 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.635755 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl"] Nov 01 16:12:24.682344 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.682314 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:24.682505 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.682370 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:12:24.719856 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.719822 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brxsz\" (UniqueName: \"kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.720000 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.719879 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.720000 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.719898 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.720608 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.720586 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.722162 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.722146 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.776462 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.776434 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brxsz\" (UniqueName: \"kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz\") pod \"maas-api-revoke-sub-d9072d42-6tlpl\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:24.893184 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:24.893101 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:25.264124 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:25.264092 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl"] Nov 01 16:12:25.265417 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:12:25.265392 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89445aed_6806_4700_b764_cc3d4dc51275.slice/crio-f37e9c89f9b056c9e95cdef1399a129b7deccdaf9d52b65291e8cee0d42a439c WatchSource:0}: Error finding container f37e9c89f9b056c9e95cdef1399a129b7deccdaf9d52b65291e8cee0d42a439c: Status 404 returned error can't find the container with id f37e9c89f9b056c9e95cdef1399a129b7deccdaf9d52b65291e8cee0d42a439c Nov 01 16:12:25.495037 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:25.494987 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" event={"ID":"89445aed-6806-4700-b764-cc3d4dc51275","Type":"ContainerStarted","Data":"f37e9c89f9b056c9e95cdef1399a129b7deccdaf9d52b65291e8cee0d42a439c"} Nov 01 16:12:26.501692 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:26.501606 5473 generic.go:356] "Generic (PLEG): container finished" podID="89445aed-6806-4700-b764-cc3d4dc51275" containerID="d187d1ad9529383a762d7362cb32e0c07e0ff397ddfc906cbe7d3e1ea8e5f8be" exitCode=0 Nov 01 16:12:26.501692 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:26.501659 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" event={"ID":"89445aed-6806-4700-b764-cc3d4dc51275","Type":"ContainerDied","Data":"d187d1ad9529383a762d7362cb32e0c07e0ff397ddfc906cbe7d3e1ea8e5f8be"} Nov 01 16:12:27.640057 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.640029 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:27.749688 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.749653 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca\") pod \"89445aed-6806-4700-b764-cc3d4dc51275\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " Nov 01 16:12:27.749910 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.749718 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz\" (UniqueName: \"kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz\") pod \"89445aed-6806-4700-b764-cc3d4dc51275\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " Nov 01 16:12:27.749910 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.749788 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca\") pod \"89445aed-6806-4700-b764-cc3d4dc51275\" (UID: \"89445aed-6806-4700-b764-cc3d4dc51275\") " Nov 01 16:12:27.750075 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.750051 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca" pod "89445aed-6806-4700-b764-cc3d4dc51275" (UID: "89445aed-6806-4700-b764-cc3d4dc51275"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:27.751864 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.751843 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca" pod "89445aed-6806-4700-b764-cc3d4dc51275" (UID: "89445aed-6806-4700-b764-cc3d4dc51275"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:27.751988 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.751967 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz" pod "89445aed-6806-4700-b764-cc3d4dc51275" (UID: "89445aed-6806-4700-b764-cc3d4dc51275"). InnerVolumeSpecName "kube-api-access-brxsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:27.851286 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.851251 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.851286 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.851281 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brxsz\" (UniqueName: \"kubernetes.io/projected/89445aed-6806-4700-b764-cc3d4dc51275-kube-api-access-brxsz\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:27.851286 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:27.851293 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89445aed-6806-4700-b764-cc3d4dc51275-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:28.513984 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:28.513953 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" Nov 01 16:12:28.513984 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:28.513962 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9072d42-6tlpl" event={"ID":"89445aed-6806-4700-b764-cc3d4dc51275","Type":"ContainerDied","Data":"f37e9c89f9b056c9e95cdef1399a129b7deccdaf9d52b65291e8cee0d42a439c"} Nov 01 16:12:28.514238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:28.514014 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37e9c89f9b056c9e95cdef1399a129b7deccdaf9d52b65291e8cee0d42a439c" Nov 01 16:12:31.110778 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:12:31.110741 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:12:35.577891 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.577853 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj"] Nov 01 16:12:35.578551 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.578530 5473 cpu_manager.go:414] "containerMap: removing container" podUID="89445aed-6806-4700-b764-cc3d4dc51275" containerName="revoke-keys" Nov 01 16:12:35.578608 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.578554 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89445aed-6806-4700-b764-cc3d4dc51275" containerName="revoke-keys" Nov 01 16:12:35.578700 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.578688 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="89445aed-6806-4700-b764-cc3d4dc51275" containerName="revoke-keys" Nov 01 16:12:35.582282 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.582265 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.635920 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.635885 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj"] Nov 01 16:12:35.682286 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.682210 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:35.682477 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.682314 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:12:35.725935 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.725906 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.726071 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.725972 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrl5\" (UniqueName: \"kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.726071 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.726042 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.827293 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.827255 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrl5\" (UniqueName: \"kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.827293 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.827296 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.827558 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.827370 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.828025 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.827956 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.829656 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.829628 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.883608 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.883579 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrl5\" (UniqueName: \"kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5\") pod \"maas-api-revoke-sub-4ff0dc45-lslfj\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:35.895698 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:35.895674 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:36.268644 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:36.268613 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj"] Nov 01 16:12:36.269278 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:12:36.269251 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c89ea00_7bde_4818_986c_45ab75ee726a.slice/crio-65b3a997fe82ee10d3141890b3b5d0c39d521b0de8ec847327ed5c9a68ff1592 WatchSource:0}: Error finding container 65b3a997fe82ee10d3141890b3b5d0c39d521b0de8ec847327ed5c9a68ff1592: Status 404 returned error can't find the container with id 65b3a997fe82ee10d3141890b3b5d0c39d521b0de8ec847327ed5c9a68ff1592 Nov 01 16:12:36.561204 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:36.561112 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" event={"ID":"4c89ea00-7bde-4818-986c-45ab75ee726a","Type":"ContainerStarted","Data":"31aed20d0aa387e2d755405eb2b4abf8355b012d14ca416f45defeb1b9781b57"} Nov 01 16:12:36.561204 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:36.561161 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" event={"ID":"4c89ea00-7bde-4818-986c-45ab75ee726a","Type":"ContainerStarted","Data":"65b3a997fe82ee10d3141890b3b5d0c39d521b0de8ec847327ed5c9a68ff1592"} Nov 01 16:12:36.725022 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:36.724967 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" podStartSLOduration=1.7249528600000001 podStartE2EDuration="1.72495286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:36.673733398 +0000 UTC m=+6895.731885248" watchObservedRunningTime="2026-11-01 16:12:36.72495286 +0000 UTC m=+6895.783104709" Nov 01 16:12:37.567868 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:37.567834 5473 generic.go:356] "Generic (PLEG): container finished" podID="4c89ea00-7bde-4818-986c-45ab75ee726a" containerID="31aed20d0aa387e2d755405eb2b4abf8355b012d14ca416f45defeb1b9781b57" exitCode=0 Nov 01 16:12:37.568064 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:37.567912 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" event={"ID":"4c89ea00-7bde-4818-986c-45ab75ee726a","Type":"ContainerDied","Data":"31aed20d0aa387e2d755405eb2b4abf8355b012d14ca416f45defeb1b9781b57"} Nov 01 16:12:38.700464 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.700439 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:38.855661 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.855552 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca\") pod \"4c89ea00-7bde-4818-986c-45ab75ee726a\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " Nov 01 16:12:38.855661 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.855644 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca\") pod \"4c89ea00-7bde-4818-986c-45ab75ee726a\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " Nov 01 16:12:38.855920 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.855678 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5\" (UniqueName: \"kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5\") pod \"4c89ea00-7bde-4818-986c-45ab75ee726a\" (UID: \"4c89ea00-7bde-4818-986c-45ab75ee726a\") " Nov 01 16:12:38.856033 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.856006 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca" pod "4c89ea00-7bde-4818-986c-45ab75ee726a" (UID: "4c89ea00-7bde-4818-986c-45ab75ee726a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:38.857835 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.857800 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca" pod "4c89ea00-7bde-4818-986c-45ab75ee726a" (UID: "4c89ea00-7bde-4818-986c-45ab75ee726a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:38.857980 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.857889 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5" pod "4c89ea00-7bde-4818-986c-45ab75ee726a" (UID: "4c89ea00-7bde-4818-986c-45ab75ee726a"). InnerVolumeSpecName "kube-api-access-gvrl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:38.956933 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.956898 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvrl5\" (UniqueName: \"kubernetes.io/projected/4c89ea00-7bde-4818-986c-45ab75ee726a-kube-api-access-gvrl5\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:38.956933 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.956927 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:38.956933 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:38.956939 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c89ea00-7bde-4818-986c-45ab75ee726a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:39.583946 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:39.583907 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" Nov 01 16:12:39.584190 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:39.583907 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff0dc45-lslfj" event={"ID":"4c89ea00-7bde-4818-986c-45ab75ee726a","Type":"ContainerDied","Data":"65b3a997fe82ee10d3141890b3b5d0c39d521b0de8ec847327ed5c9a68ff1592"} Nov 01 16:12:39.584190 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:39.584058 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b3a997fe82ee10d3141890b3b5d0c39d521b0de8ec847327ed5c9a68ff1592" Nov 01 16:12:44.977403 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:44.977374 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:12:45.006765 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:45.006742 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:12:46.129522 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.129491 5473 scope.go:122] "RemoveContainer" containerID="91e5fb7da76f0e8ad2200127d173cb0d4d02a1edd3ef37809afb31b5997416e8" Nov 01 16:12:46.226704 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.226671 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w"] Nov 01 16:12:46.227089 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.227076 5473 cpu_manager.go:414] "containerMap: removing container" podUID="4c89ea00-7bde-4818-986c-45ab75ee726a" containerName="revoke-keys" Nov 01 16:12:46.227139 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.227091 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c89ea00-7bde-4818-986c-45ab75ee726a" containerName="revoke-keys" Nov 01 16:12:46.227179 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.227163 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c89ea00-7bde-4818-986c-45ab75ee726a" containerName="revoke-keys" Nov 01 16:12:46.231360 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.231342 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.284497 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.284455 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w"] Nov 01 16:12:46.294543 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.294511 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8k9c\" (UniqueName: \"kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.294684 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.294625 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.294684 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.294676 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.331309 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.331280 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:46.331431 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.331382 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:12:46.395527 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.395465 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.395527 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.395511 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.395695 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.395556 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8k9c\" (UniqueName: \"kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.396198 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.396169 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.397754 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.397732 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.451697 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.451674 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8k9c\" (UniqueName: \"kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c\") pod \"maas-api-revoke-sub-7f8cd973-62g5w\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.544699 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.544671 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:46.915384 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:46.915358 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w"] Nov 01 16:12:46.916822 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:12:46.916793 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d435d7_79ab_4e9c_b1ff_555cfd6ba1f2.slice/crio-cf92ee5f9b5c1e107168e7c96d84ce342b04b425ff9dc057db6cd4b86cb2abdb WatchSource:0}: Error finding container cf92ee5f9b5c1e107168e7c96d84ce342b04b425ff9dc057db6cd4b86cb2abdb: Status 404 returned error can't find the container with id cf92ee5f9b5c1e107168e7c96d84ce342b04b425ff9dc057db6cd4b86cb2abdb Nov 01 16:12:47.659880 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:47.659841 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" event={"ID":"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2","Type":"ContainerStarted","Data":"86f3846d55bf3ce37e85a9b87c8906527a42428de1ced69fdd11daf090d44291"} Nov 01 16:12:47.660323 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:47.659893 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" event={"ID":"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2","Type":"ContainerStarted","Data":"cf92ee5f9b5c1e107168e7c96d84ce342b04b425ff9dc057db6cd4b86cb2abdb"} Nov 01 16:12:47.776091 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:47.776038 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" podStartSLOduration=1.7760221440000001 podStartE2EDuration="1.776022144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:47.773437053 +0000 UTC m=+6906.831588918" watchObservedRunningTime="2026-11-01 16:12:47.776022144 +0000 UTC m=+6906.834173998" Nov 01 16:12:48.666768 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:48.666729 5473 generic.go:356] "Generic (PLEG): container finished" podID="73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" containerID="86f3846d55bf3ce37e85a9b87c8906527a42428de1ced69fdd11daf090d44291" exitCode=0 Nov 01 16:12:48.667205 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:48.666803 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" event={"ID":"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2","Type":"ContainerDied","Data":"86f3846d55bf3ce37e85a9b87c8906527a42428de1ced69fdd11daf090d44291"} Nov 01 16:12:49.806606 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.806582 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:49.825643 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.825615 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca\") pod \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " Nov 01 16:12:49.825757 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.825652 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca\") pod \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " Nov 01 16:12:49.825757 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.825733 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c\" (UniqueName: \"kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c\") pod \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\" (UID: \"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2\") " Nov 01 16:12:49.825991 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.825967 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca" pod "73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" (UID: "73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:49.827658 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.827636 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca" pod "73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" (UID: "73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:49.827855 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.827834 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c" pod "73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" (UID: "73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2"). InnerVolumeSpecName "kube-api-access-p8k9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:49.926729 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.926649 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8k9c\" (UniqueName: \"kubernetes.io/projected/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-kube-api-access-p8k9c\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:49.926729 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.926679 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:49.926729 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:49.926690 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:50.679510 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:50.679479 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" Nov 01 16:12:50.679510 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:50.679494 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f8cd973-62g5w" event={"ID":"73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2","Type":"ContainerDied","Data":"cf92ee5f9b5c1e107168e7c96d84ce342b04b425ff9dc057db6cd4b86cb2abdb"} Nov 01 16:12:50.679731 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:50.679539 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf92ee5f9b5c1e107168e7c96d84ce342b04b425ff9dc057db6cd4b86cb2abdb" Nov 01 16:12:57.168734 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.168698 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp"] Nov 01 16:12:57.169164 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.169150 5473 cpu_manager.go:414] "containerMap: removing container" podUID="73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" containerName="revoke-keys" Nov 01 16:12:57.169236 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.169165 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" containerName="revoke-keys" Nov 01 16:12:57.169291 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.169280 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2" containerName="revoke-keys" Nov 01 16:12:57.172543 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.172526 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.225545 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.225508 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp"] Nov 01 16:12:57.271990 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.271954 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:57.272151 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.272080 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:12:57.300987 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.300960 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qltm\" (UniqueName: \"kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.301155 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.301010 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.301155 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.301095 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.402454 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.402424 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qltm\" (UniqueName: \"kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.402608 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.402478 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.402608 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.402494 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.403204 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.403185 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.404892 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.404875 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.459750 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.459691 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qltm\" (UniqueName: \"kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm\") pod \"maas-api-revoke-sub-b12f4962-ps7fp\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.488791 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.488765 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:12:57.861429 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:57.861404 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp"] Nov 01 16:12:57.863058 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:12:57.863032 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6a1fce_1ee4_4564_9922_daaed112fcfc.slice/crio-f1300a0e7189fdfbca6444bcff54758e26283eb1c410ce25d3a1de28c1651e0f WatchSource:0}: Error finding container f1300a0e7189fdfbca6444bcff54758e26283eb1c410ce25d3a1de28c1651e0f: Status 404 returned error can't find the container with id f1300a0e7189fdfbca6444bcff54758e26283eb1c410ce25d3a1de28c1651e0f Nov 01 16:12:58.047247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:58.047199 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk"] Nov 01 16:12:58.049647 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:58.049623 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vjqqk"] Nov 01 16:12:58.724427 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:58.724390 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" event={"ID":"3d6a1fce-1ee4-4564-9922-daaed112fcfc","Type":"ContainerStarted","Data":"6a6263c2e817946fd4d314543fa1bd721b1d8221e57fce602da281b85bfd3e0a"} Nov 01 16:12:58.724910 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:58.724444 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" event={"ID":"3d6a1fce-1ee4-4564-9922-daaed112fcfc","Type":"ContainerStarted","Data":"f1300a0e7189fdfbca6444bcff54758e26283eb1c410ce25d3a1de28c1651e0f"} Nov 01 16:12:58.839859 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:58.839812 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" podStartSLOduration=1.8397977760000002 podStartE2EDuration="1.839797776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:58.837946771 +0000 UTC m=+6917.896098621" watchObservedRunningTime="2026-11-01 16:12:58.839797776 +0000 UTC m=+6917.897949626" Nov 01 16:12:59.080842 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:59.080809 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f1435e-1c24-429c-9147-c14332d7737c" path="/var/lib/kubelet/pods/47f1435e-1c24-429c-9147-c14332d7737c/volumes" Nov 01 16:12:59.735249 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:59.735103 5473 generic.go:356] "Generic (PLEG): container finished" podID="3d6a1fce-1ee4-4564-9922-daaed112fcfc" containerID="6a6263c2e817946fd4d314543fa1bd721b1d8221e57fce602da281b85bfd3e0a" exitCode=0 Nov 01 16:12:59.735249 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:12:59.735165 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" event={"ID":"3d6a1fce-1ee4-4564-9922-daaed112fcfc","Type":"ContainerDied","Data":"6a6263c2e817946fd4d314543fa1bd721b1d8221e57fce602da281b85bfd3e0a"} Nov 01 16:13:01.373290 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.349396 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:13:01.382238 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:13:01.374446 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:13:01.506612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.506581 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm\" (UniqueName: \"kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm\") pod \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " Nov 01 16:13:01.510512 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.510488 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca\") pod \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " Nov 01 16:13:01.524764 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.524734 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca" pod "3d6a1fce-1ee4-4564-9922-daaed112fcfc" (UID: "3d6a1fce-1ee4-4564-9922-daaed112fcfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:01.566311 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.518537 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca\") pod \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\" (UID: \"3d6a1fce-1ee4-4564-9922-daaed112fcfc\") " Nov 01 16:13:01.566968 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.566938 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:01.572370 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.572338 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm" pod "3d6a1fce-1ee4-4564-9922-daaed112fcfc" (UID: "3d6a1fce-1ee4-4564-9922-daaed112fcfc"). InnerVolumeSpecName "kube-api-access-5qltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:01.599241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.596333 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca" pod "3d6a1fce-1ee4-4564-9922-daaed112fcfc" (UID: "3d6a1fce-1ee4-4564-9922-daaed112fcfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:01.669925 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.669836 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qltm\" (UniqueName: \"kubernetes.io/projected/3d6a1fce-1ee4-4564-9922-daaed112fcfc-kube-api-access-5qltm\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:01.670148 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.670129 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d6a1fce-1ee4-4564-9922-daaed112fcfc-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:01.779629 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.779591 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" event={"ID":"3d6a1fce-1ee4-4564-9922-daaed112fcfc","Type":"ContainerDied","Data":"f1300a0e7189fdfbca6444bcff54758e26283eb1c410ce25d3a1de28c1651e0f"} Nov 01 16:13:01.779876 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.779858 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1300a0e7189fdfbca6444bcff54758e26283eb1c410ce25d3a1de28c1651e0f" Nov 01 16:13:01.780085 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:01.780070 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b12f4962-ps7fp" Nov 01 16:13:07.873484 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.873448 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj"] Nov 01 16:13:07.873965 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.873876 5473 cpu_manager.go:414] "containerMap: removing container" podUID="3d6a1fce-1ee4-4564-9922-daaed112fcfc" containerName="revoke-keys" Nov 01 16:13:07.873965 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.873887 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6a1fce-1ee4-4564-9922-daaed112fcfc" containerName="revoke-keys" Nov 01 16:13:07.873965 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.873952 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6a1fce-1ee4-4564-9922-daaed112fcfc" containerName="revoke-keys" Nov 01 16:13:07.877051 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.877036 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:07.929881 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.929850 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:13:07.929881 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.929855 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj"] Nov 01 16:13:07.930130 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.929961 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:13:07.947492 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.947467 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gcjl\" (UniqueName: \"kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:07.947691 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.947512 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:07.947691 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:07.947644 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.048735 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.048692 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.048945 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.048812 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.048945 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.048909 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gcjl\" (UniqueName: \"kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.049467 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.049443 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.051102 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.051083 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.105173 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.105148 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gcjl\" (UniqueName: \"kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl\") pod \"maas-api-revoke-sub-b5aa1b67-dj2cj\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.189401 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.189320 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:08.561192 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.561165 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj"] Nov 01 16:13:08.562010 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:13:08.561985 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aefbd7_a0da_4825_8613_1f5661149cff.slice/crio-66c4dd3490c1b5bb8cfd4a598842f3b69117fdc86cdcc565bf651e7e31c618ab WatchSource:0}: Error finding container 66c4dd3490c1b5bb8cfd4a598842f3b69117fdc86cdcc565bf651e7e31c618ab: Status 404 returned error can't find the container with id 66c4dd3490c1b5bb8cfd4a598842f3b69117fdc86cdcc565bf651e7e31c618ab Nov 01 16:13:08.820730 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:08.820637 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" event={"ID":"97aefbd7-a0da-4825-8613-1f5661149cff","Type":"ContainerStarted","Data":"66c4dd3490c1b5bb8cfd4a598842f3b69117fdc86cdcc565bf651e7e31c618ab"} Nov 01 16:13:09.408891 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:13:09.408860 5473 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aefbd7_a0da_4825_8613_1f5661149cff.slice/crio-conmon-72b2bdfffed6a58cfee44a3c64e8520ea6a88589a127b4453e909b7886f98b8b.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:13:09.828023 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:09.827979 5473 generic.go:356] "Generic (PLEG): container finished" podID="97aefbd7-a0da-4825-8613-1f5661149cff" containerID="72b2bdfffed6a58cfee44a3c64e8520ea6a88589a127b4453e909b7886f98b8b" exitCode=0 Nov 01 16:13:09.828208 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:09.828073 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" event={"ID":"97aefbd7-a0da-4825-8613-1f5661149cff","Type":"ContainerDied","Data":"72b2bdfffed6a58cfee44a3c64e8520ea6a88589a127b4453e909b7886f98b8b"} Nov 01 16:13:11.027813 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.027781 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:11.190310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.188527 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca\") pod \"97aefbd7-a0da-4825-8613-1f5661149cff\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " Nov 01 16:13:11.190310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.188619 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl\" (UniqueName: \"kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl\") pod \"97aefbd7-a0da-4825-8613-1f5661149cff\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " Nov 01 16:13:11.190310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.188738 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca\") pod \"97aefbd7-a0da-4825-8613-1f5661149cff\" (UID: \"97aefbd7-a0da-4825-8613-1f5661149cff\") " Nov 01 16:13:11.207240 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.206431 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca" pod "97aefbd7-a0da-4825-8613-1f5661149cff" (UID: "97aefbd7-a0da-4825-8613-1f5661149cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:11.222241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.219185 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca" pod "97aefbd7-a0da-4825-8613-1f5661149cff" (UID: "97aefbd7-a0da-4825-8613-1f5661149cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:11.288477 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.288406 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl" pod "97aefbd7-a0da-4825-8613-1f5661149cff" (UID: "97aefbd7-a0da-4825-8613-1f5661149cff"). InnerVolumeSpecName "kube-api-access-6gcjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:11.292013 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.291922 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:11.292013 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.291960 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aefbd7-a0da-4825-8613-1f5661149cff-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:11.292013 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.291978 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gcjl\" (UniqueName: \"kubernetes.io/projected/97aefbd7-a0da-4825-8613-1f5661149cff-kube-api-access-6gcjl\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:11.864250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.861363 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" event={"ID":"97aefbd7-a0da-4825-8613-1f5661149cff","Type":"ContainerDied","Data":"66c4dd3490c1b5bb8cfd4a598842f3b69117fdc86cdcc565bf651e7e31c618ab"} Nov 01 16:13:11.864250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.861431 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66c4dd3490c1b5bb8cfd4a598842f3b69117fdc86cdcc565bf651e7e31c618ab" Nov 01 16:13:11.864250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:11.861509 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5aa1b67-dj2cj" Nov 01 16:13:12.581243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.580383 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-p8scg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:12.581243 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.580464 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-p8scg" podUID="552f992f-89c3-43bf-8a2c-44665d612997" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:12.605238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.604366 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-56jt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:12.605238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.604427 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-56jt9" podUID="8a84ef62-7215-4e81-8585-2ed334e28a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:12.645371 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.645332 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-l5vkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:12.645686 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.645642 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-l5vkl" podUID="aea2e96a-5411-4745-903f-275b69c10843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:12.864482 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.864291 5473 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-x769v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:12.864482 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:12.864358 5473 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-x769v" podUID="bdad6cc9-742d-42e6-a82d-b8c8664ed362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:18.819171 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.819127 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd"] Nov 01 16:13:18.819873 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.819851 5473 cpu_manager.go:414] "containerMap: removing container" podUID="97aefbd7-a0da-4825-8613-1f5661149cff" containerName="revoke-keys" Nov 01 16:13:18.819920 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.819877 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97aefbd7-a0da-4825-8613-1f5661149cff" containerName="revoke-keys" Nov 01 16:13:18.820031 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.820017 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="97aefbd7-a0da-4825-8613-1f5661149cff" containerName="revoke-keys" Nov 01 16:13:18.824046 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.824025 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:18.874133 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.874094 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd"] Nov 01 16:13:18.912811 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.912783 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:18.913007 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.912834 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkcx\" (UniqueName: \"kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:18.913007 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.912854 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:18.920368 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.920343 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:13:18.920476 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:18.920394 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:13:19.014477 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.014446 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.014672 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.014497 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkcx\" (UniqueName: \"kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.014672 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.014515 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.015166 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.015140 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.016885 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.016867 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.051112 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.051085 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql"] Nov 01 16:13:19.053511 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.053489 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-8gvql"] Nov 01 16:13:19.070873 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.070823 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkcx\" (UniqueName: \"kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx\") pod \"maas-api-revoke-sub-2d87a446-9x6gd\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.080682 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.080660 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243fd851-1e2a-47a3-bd76-ba213767796a" path="/var/lib/kubelet/pods/243fd851-1e2a-47a3-bd76-ba213767796a/volumes" Nov 01 16:13:19.136861 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.136828 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:19.510805 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.510779 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd"] Nov 01 16:13:19.511514 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:13:19.511491 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cd8cc4_796d_421c_92aa_81cadff0d454.slice/crio-acccb7117a32ab66e75409ed7dbf9d363fdbffafb98eccfdae1d1b080b9a91a7 WatchSource:0}: Error finding container acccb7117a32ab66e75409ed7dbf9d363fdbffafb98eccfdae1d1b080b9a91a7: Status 404 returned error can't find the container with id acccb7117a32ab66e75409ed7dbf9d363fdbffafb98eccfdae1d1b080b9a91a7 Nov 01 16:13:19.940030 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.939991 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" event={"ID":"06cd8cc4-796d-421c-92aa-81cadff0d454","Type":"ContainerStarted","Data":"0a758fe65440a4f734145e60cb67637d0a4ae0eb715ba775ba872b927322d1d5"} Nov 01 16:13:19.940530 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:19.940043 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" event={"ID":"06cd8cc4-796d-421c-92aa-81cadff0d454","Type":"ContainerStarted","Data":"acccb7117a32ab66e75409ed7dbf9d363fdbffafb98eccfdae1d1b080b9a91a7"} Nov 01 16:13:20.053353 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:20.053301 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:20.103170 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:20.103076 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" podStartSLOduration=2.1030616 podStartE2EDuration="2.1030616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:20.052905315 +0000 UTC m=+6939.111057163" watchObservedRunningTime="2026-11-01 16:13:20.1030616 +0000 UTC m=+6939.161213450" Nov 01 16:13:20.953736 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:20.953700 5473 generic.go:356] "Generic (PLEG): container finished" podID="06cd8cc4-796d-421c-92aa-81cadff0d454" containerID="0a758fe65440a4f734145e60cb67637d0a4ae0eb715ba775ba872b927322d1d5" exitCode=0 Nov 01 16:13:20.954371 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:20.953743 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" event={"ID":"06cd8cc4-796d-421c-92aa-81cadff0d454","Type":"ContainerDied","Data":"0a758fe65440a4f734145e60cb67637d0a4ae0eb715ba775ba872b927322d1d5"} Nov 01 16:13:22.717488 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.717462 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:22.822595 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.822557 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca\") pod \"06cd8cc4-796d-421c-92aa-81cadff0d454\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " Nov 01 16:13:22.822804 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.822664 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx\" (UniqueName: \"kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx\") pod \"06cd8cc4-796d-421c-92aa-81cadff0d454\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " Nov 01 16:13:22.822804 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.822720 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca\") pod \"06cd8cc4-796d-421c-92aa-81cadff0d454\" (UID: \"06cd8cc4-796d-421c-92aa-81cadff0d454\") " Nov 01 16:13:22.822964 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.822934 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca" pod "06cd8cc4-796d-421c-92aa-81cadff0d454" (UID: "06cd8cc4-796d-421c-92aa-81cadff0d454"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:22.823169 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.823132 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:22.825818 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.825794 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca" pod "06cd8cc4-796d-421c-92aa-81cadff0d454" (UID: "06cd8cc4-796d-421c-92aa-81cadff0d454"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:22.916641 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.916557 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx" pod "06cd8cc4-796d-421c-92aa-81cadff0d454" (UID: "06cd8cc4-796d-421c-92aa-81cadff0d454"). InnerVolumeSpecName "kube-api-access-stkcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:22.928242 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.924843 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stkcx\" (UniqueName: \"kubernetes.io/projected/06cd8cc4-796d-421c-92aa-81cadff0d454-kube-api-access-stkcx\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:22.928242 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:22.924878 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06cd8cc4-796d-421c-92aa-81cadff0d454-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:23.058241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:23.048555 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" event={"ID":"06cd8cc4-796d-421c-92aa-81cadff0d454","Type":"ContainerDied","Data":"acccb7117a32ab66e75409ed7dbf9d363fdbffafb98eccfdae1d1b080b9a91a7"} Nov 01 16:13:23.058579 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:23.058538 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acccb7117a32ab66e75409ed7dbf9d363fdbffafb98eccfdae1d1b080b9a91a7" Nov 01 16:13:23.058993 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:23.058797 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d87a446-9x6gd" Nov 01 16:13:29.663749 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.663711 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw"] Nov 01 16:13:29.664507 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.664483 5473 cpu_manager.go:414] "containerMap: removing container" podUID="06cd8cc4-796d-421c-92aa-81cadff0d454" containerName="revoke-keys" Nov 01 16:13:29.664567 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.664511 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06cd8cc4-796d-421c-92aa-81cadff0d454" containerName="revoke-keys" Nov 01 16:13:29.664626 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.664613 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="06cd8cc4-796d-421c-92aa-81cadff0d454" containerName="revoke-keys" Nov 01 16:13:29.668624 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.668600 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.718893 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.718863 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw"] Nov 01 16:13:29.765425 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.765396 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:13:29.765425 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.765396 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:13:29.827453 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.827423 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbg9\" (UniqueName: \"kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.827612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.827474 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.827690 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.827664 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.928458 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.928384 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.928611 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.928473 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbg9\" (UniqueName: \"kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.928611 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.928515 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.929253 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.929200 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.930776 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.930753 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:29.987111 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:29.987076 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbg9\" (UniqueName: \"kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9\") pod \"maas-api-revoke-sub-ee0d5d16-6gddw\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:30.282048 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:30.282014 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:30.652661 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:30.652629 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw"] Nov 01 16:13:30.653168 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:13:30.653143 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2521c6c_10d1_4562_aff1_554407a77cca.slice/crio-dc706ea2ad32b3baeb9eb2ae3532f83564020c1bffcac2f3dde647f63b6b3bd9 WatchSource:0}: Error finding container dc706ea2ad32b3baeb9eb2ae3532f83564020c1bffcac2f3dde647f63b6b3bd9: Status 404 returned error can't find the container with id dc706ea2ad32b3baeb9eb2ae3532f83564020c1bffcac2f3dde647f63b6b3bd9 Nov 01 16:13:31.095335 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:31.095290 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" event={"ID":"b2521c6c-10d1-4562-aff1-554407a77cca","Type":"ContainerStarted","Data":"d7d0f4291f598e8f9b89a554f1b4b58b8aa2243d0ff848e24b2288982f8397b5"} Nov 01 16:13:31.095335 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:31.095342 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" event={"ID":"b2521c6c-10d1-4562-aff1-554407a77cca","Type":"ContainerStarted","Data":"dc706ea2ad32b3baeb9eb2ae3532f83564020c1bffcac2f3dde647f63b6b3bd9"} Nov 01 16:13:31.112668 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:13:31.112633 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:13:31.209152 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:31.209051 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" podStartSLOduration=2.209035357 podStartE2EDuration="2.209035357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:31.207558594 +0000 UTC m=+6950.265710446" watchObservedRunningTime="2026-11-01 16:13:31.209035357 +0000 UTC m=+6950.267187252" Nov 01 16:13:32.106942 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:32.106897 5473 generic.go:356] "Generic (PLEG): container finished" podID="b2521c6c-10d1-4562-aff1-554407a77cca" containerID="d7d0f4291f598e8f9b89a554f1b4b58b8aa2243d0ff848e24b2288982f8397b5" exitCode=0 Nov 01 16:13:32.107517 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:32.107013 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" event={"ID":"b2521c6c-10d1-4562-aff1-554407a77cca","Type":"ContainerDied","Data":"d7d0f4291f598e8f9b89a554f1b4b58b8aa2243d0ff848e24b2288982f8397b5"} Nov 01 16:13:33.619490 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.619162 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:33.697803 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.697751 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca\") pod \"b2521c6c-10d1-4562-aff1-554407a77cca\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " Nov 01 16:13:33.698619 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.698582 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9\" (UniqueName: \"kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9\") pod \"b2521c6c-10d1-4562-aff1-554407a77cca\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " Nov 01 16:13:33.698844 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.698816 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca\") pod \"b2521c6c-10d1-4562-aff1-554407a77cca\" (UID: \"b2521c6c-10d1-4562-aff1-554407a77cca\") " Nov 01 16:13:33.699543 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.698507 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca" pod "b2521c6c-10d1-4562-aff1-554407a77cca" (UID: "b2521c6c-10d1-4562-aff1-554407a77cca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:33.709252 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.708861 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9" pod "b2521c6c-10d1-4562-aff1-554407a77cca" (UID: "b2521c6c-10d1-4562-aff1-554407a77cca"). InnerVolumeSpecName "kube-api-access-tsbg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:33.722395 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.722346 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca" pod "b2521c6c-10d1-4562-aff1-554407a77cca" (UID: "b2521c6c-10d1-4562-aff1-554407a77cca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:33.826296 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.801894 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:33.826296 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.801936 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbg9\" (UniqueName: \"kubernetes.io/projected/b2521c6c-10d1-4562-aff1-554407a77cca-kube-api-access-tsbg9\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:33.826296 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:33.801954 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2521c6c-10d1-4562-aff1-554407a77cca-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:34.151901 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:34.151805 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" event={"ID":"b2521c6c-10d1-4562-aff1-554407a77cca","Type":"ContainerDied","Data":"dc706ea2ad32b3baeb9eb2ae3532f83564020c1bffcac2f3dde647f63b6b3bd9"} Nov 01 16:13:34.152187 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:34.152165 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc706ea2ad32b3baeb9eb2ae3532f83564020c1bffcac2f3dde647f63b6b3bd9" Nov 01 16:13:34.152415 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:34.152399 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee0d5d16-6gddw" Nov 01 16:13:35.631875 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:35.631845 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:40.433267 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.433233 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj"] Nov 01 16:13:40.433759 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.433655 5473 cpu_manager.go:414] "containerMap: removing container" podUID="b2521c6c-10d1-4562-aff1-554407a77cca" containerName="revoke-keys" Nov 01 16:13:40.433759 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.433667 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2521c6c-10d1-4562-aff1-554407a77cca" containerName="revoke-keys" Nov 01 16:13:40.433759 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.433730 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2521c6c-10d1-4562-aff1-554407a77cca" containerName="revoke-keys" Nov 01 16:13:40.436868 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.436852 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.488401 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.488363 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.488614 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.488473 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4whc\" (UniqueName: \"kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.488614 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.488538 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.490329 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.490297 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj"] Nov 01 16:13:40.536503 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.536471 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:13:40.536668 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.536565 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:13:40.589314 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.589277 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.589498 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.589377 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.589498 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.589427 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4whc\" (UniqueName: \"kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.590064 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.590037 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.591626 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.591606 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.646496 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.646471 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4whc\" (UniqueName: \"kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc\") pod \"maas-api-revoke-sub-206f88f6-qq7jj\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:40.749762 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:40.749698 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:41.120271 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:41.120242 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj"] Nov 01 16:13:41.121343 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:13:41.121318 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d02152c_6936_4a63_a7c5_312f761c81ef.slice/crio-e13cd97aed4249c09c4459fdbc733e4591567770632d57881aa16769f72e9829 WatchSource:0}: Error finding container e13cd97aed4249c09c4459fdbc733e4591567770632d57881aa16769f72e9829: Status 404 returned error can't find the container with id e13cd97aed4249c09c4459fdbc733e4591567770632d57881aa16769f72e9829 Nov 01 16:13:41.192433 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:41.192402 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" event={"ID":"1d02152c-6936-4a63-a7c5-312f761c81ef","Type":"ContainerStarted","Data":"e13cd97aed4249c09c4459fdbc733e4591567770632d57881aa16769f72e9829"} Nov 01 16:13:42.200156 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:42.200119 5473 generic.go:356] "Generic (PLEG): container finished" podID="1d02152c-6936-4a63-a7c5-312f761c81ef" containerID="3591c2d104d28b398db17cb73b41a6610621643023385cd51fc1042aea2ba8ef" exitCode=0 Nov 01 16:13:42.200566 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:42.200169 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" event={"ID":"1d02152c-6936-4a63-a7c5-312f761c81ef","Type":"ContainerDied","Data":"3591c2d104d28b398db17cb73b41a6610621643023385cd51fc1042aea2ba8ef"} Nov 01 16:13:43.417091 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.416734 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:43.523636 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.523597 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc\" (UniqueName: \"kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc\") pod \"1d02152c-6936-4a63-a7c5-312f761c81ef\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " Nov 01 16:13:43.523869 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.523724 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca\") pod \"1d02152c-6936-4a63-a7c5-312f761c81ef\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " Nov 01 16:13:43.523869 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.523817 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca\") pod \"1d02152c-6936-4a63-a7c5-312f761c81ef\" (UID: \"1d02152c-6936-4a63-a7c5-312f761c81ef\") " Nov 01 16:13:43.524236 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.524184 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca" pod "1d02152c-6936-4a63-a7c5-312f761c81ef" (UID: "1d02152c-6936-4a63-a7c5-312f761c81ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:43.526507 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.526474 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc" pod "1d02152c-6936-4a63-a7c5-312f761c81ef" (UID: "1d02152c-6936-4a63-a7c5-312f761c81ef"). InnerVolumeSpecName "kube-api-access-f4whc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:43.544782 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.544743 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca" pod "1d02152c-6936-4a63-a7c5-312f761c81ef" (UID: "1d02152c-6936-4a63-a7c5-312f761c81ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:43.634019 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.633925 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4whc\" (UniqueName: \"kubernetes.io/projected/1d02152c-6936-4a63-a7c5-312f761c81ef-kube-api-access-f4whc\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:43.634019 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.633970 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:43.634019 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:43.633987 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d02152c-6936-4a63-a7c5-312f761c81ef-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:44.242538 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:44.242497 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" event={"ID":"1d02152c-6936-4a63-a7c5-312f761c81ef","Type":"ContainerDied","Data":"e13cd97aed4249c09c4459fdbc733e4591567770632d57881aa16769f72e9829"} Nov 01 16:13:44.242836 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:44.242815 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e13cd97aed4249c09c4459fdbc733e4591567770632d57881aa16769f72e9829" Nov 01 16:13:44.243029 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:44.243014 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f88f6-qq7jj" Nov 01 16:13:51.155451 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.155415 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx"] Nov 01 16:13:51.155858 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.155813 5473 cpu_manager.go:414] "containerMap: removing container" podUID="1d02152c-6936-4a63-a7c5-312f761c81ef" containerName="revoke-keys" Nov 01 16:13:51.155858 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.155823 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d02152c-6936-4a63-a7c5-312f761c81ef" containerName="revoke-keys" Nov 01 16:13:51.156006 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.155993 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d02152c-6936-4a63-a7c5-312f761c81ef" containerName="revoke-keys" Nov 01 16:13:51.160425 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.160407 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.211065 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.211025 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx"] Nov 01 16:13:51.257603 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.257566 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:13:51.257799 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.257607 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:13:51.337586 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.337550 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.337786 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.337598 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6kf\" (UniqueName: \"kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.337786 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.337708 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.438696 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.438613 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.438696 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.438657 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6kf\" (UniqueName: \"kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.438947 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.438715 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.439289 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.439270 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.440950 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.440926 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.495234 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.495192 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6kf\" (UniqueName: \"kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf\") pod \"maas-api-revoke-sub-1d74f66d-g52kx\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:51.773144 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:51.773116 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:52.143703 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:52.143674 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx"] Nov 01 16:13:52.144462 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:13:52.144429 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e84a7f2_5820_42c9_8adc_063afa5a3e0a.slice/crio-cebdccbb0de57025e22ffcb72e2f20b1572604c3f27bd24fe0d8effbeca21527 WatchSource:0}: Error finding container cebdccbb0de57025e22ffcb72e2f20b1572604c3f27bd24fe0d8effbeca21527: Status 404 returned error can't find the container with id cebdccbb0de57025e22ffcb72e2f20b1572604c3f27bd24fe0d8effbeca21527 Nov 01 16:13:52.302702 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:52.302664 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" event={"ID":"2e84a7f2-5820-42c9-8adc-063afa5a3e0a","Type":"ContainerStarted","Data":"cebdccbb0de57025e22ffcb72e2f20b1572604c3f27bd24fe0d8effbeca21527"} Nov 01 16:13:53.322502 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:53.322449 5473 generic.go:356] "Generic (PLEG): container finished" podID="2e84a7f2-5820-42c9-8adc-063afa5a3e0a" containerID="8dec4be9fe21bc0245a28356ed988e0864c508205a36a2f68848b139ec303cdb" exitCode=0 Nov 01 16:13:53.323019 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:53.322517 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" event={"ID":"2e84a7f2-5820-42c9-8adc-063afa5a3e0a","Type":"ContainerDied","Data":"8dec4be9fe21bc0245a28356ed988e0864c508205a36a2f68848b139ec303cdb"} Nov 01 16:13:55.049480 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.049446 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:55.107942 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.106732 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca\") pod \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " Nov 01 16:13:55.107942 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.106856 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca\") pod \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " Nov 01 16:13:55.107942 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.106918 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf\" (UniqueName: \"kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf\") pod \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\" (UID: \"2e84a7f2-5820-42c9-8adc-063afa5a3e0a\") " Nov 01 16:13:55.118471 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.118418 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca" pod "2e84a7f2-5820-42c9-8adc-063afa5a3e0a" (UID: "2e84a7f2-5820-42c9-8adc-063afa5a3e0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:55.147245 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.144731 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca" pod "2e84a7f2-5820-42c9-8adc-063afa5a3e0a" (UID: "2e84a7f2-5820-42c9-8adc-063afa5a3e0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:13:55.147245 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.147090 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf" pod "2e84a7f2-5820-42c9-8adc-063afa5a3e0a" (UID: "2e84a7f2-5820-42c9-8adc-063afa5a3e0a"). InnerVolumeSpecName "kube-api-access-pc6kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:55.217250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.209959 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:55.217250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.209997 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:55.217250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.210017 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6kf\" (UniqueName: \"kubernetes.io/projected/2e84a7f2-5820-42c9-8adc-063afa5a3e0a-kube-api-access-pc6kf\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:55.450828 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.450728 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" event={"ID":"2e84a7f2-5820-42c9-8adc-063afa5a3e0a","Type":"ContainerDied","Data":"cebdccbb0de57025e22ffcb72e2f20b1572604c3f27bd24fe0d8effbeca21527"} Nov 01 16:13:55.451156 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.451136 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cebdccbb0de57025e22ffcb72e2f20b1572604c3f27bd24fe0d8effbeca21527" Nov 01 16:13:55.451376 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:55.451077 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d74f66d-g52kx" Nov 01 16:13:58.048544 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:58.048510 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp"] Nov 01 16:13:58.098390 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:58.098356 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d855e84b-jmtnp"] Nov 01 16:13:59.080673 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:13:59.080636 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060e2d94-0ea3-48cf-975d-17f0076df73a" path="/var/lib/kubelet/pods/060e2d94-0ea3-48cf-975d-17f0076df73a/volumes" Nov 01 16:14:01.110854 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:14:01.110811 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:14:01.859808 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.859772 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml"] Nov 01 16:14:01.860250 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.860235 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2e84a7f2-5820-42c9-8adc-063afa5a3e0a" containerName="revoke-keys" Nov 01 16:14:01.860330 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.860251 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e84a7f2-5820-42c9-8adc-063afa5a3e0a" containerName="revoke-keys" Nov 01 16:14:01.860330 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.860321 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e84a7f2-5820-42c9-8adc-063afa5a3e0a" containerName="revoke-keys" Nov 01 16:14:01.863968 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.863950 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.872020 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.871939 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.872020 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.871977 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgjh\" (UniqueName: \"kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.872201 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.872174 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.915935 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.915892 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml"] Nov 01 16:14:01.962334 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.962303 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:14:01.962501 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.962353 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:14:01.972847 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.972813 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.973011 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.972873 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.973011 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.972903 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgjh\" (UniqueName: \"kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.973533 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.973513 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:01.975294 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:01.975270 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:02.029868 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:02.029838 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgjh\" (UniqueName: \"kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh\") pod \"maas-api-revoke-sub-7fc48a7b-266ml\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:02.178146 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:02.178058 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:02.549673 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:02.549645 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml"] Nov 01 16:14:02.550502 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:14:02.550475 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b678595_2640_4558_acfa_2361f9f36dbb.slice/crio-db0aff32918c5e6898df64f834bbe500a19dd0f2f0c8468b022ac3c73389b71c WatchSource:0}: Error finding container db0aff32918c5e6898df64f834bbe500a19dd0f2f0c8468b022ac3c73389b71c: Status 404 returned error can't find the container with id db0aff32918c5e6898df64f834bbe500a19dd0f2f0c8468b022ac3c73389b71c Nov 01 16:14:03.500029 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:03.499997 5473 generic.go:356] "Generic (PLEG): container finished" podID="2b678595-2640-4558-acfa-2361f9f36dbb" containerID="8ddfcfc121a65af5d971d36019eb5d92514f1f22fc8c8b99417fc1f9f34767f3" exitCode=0 Nov 01 16:14:03.500388 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:03.500041 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" event={"ID":"2b678595-2640-4558-acfa-2361f9f36dbb","Type":"ContainerDied","Data":"8ddfcfc121a65af5d971d36019eb5d92514f1f22fc8c8b99417fc1f9f34767f3"} Nov 01 16:14:03.500388 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:03.500079 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" event={"ID":"2b678595-2640-4558-acfa-2361f9f36dbb","Type":"ContainerStarted","Data":"db0aff32918c5e6898df64f834bbe500a19dd0f2f0c8468b022ac3c73389b71c"} Nov 01 16:14:04.705021 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.704983 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:04.800905 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.800839 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh\" (UniqueName: \"kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh\") pod \"2b678595-2640-4558-acfa-2361f9f36dbb\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " Nov 01 16:14:04.801159 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.801002 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca\") pod \"2b678595-2640-4558-acfa-2361f9f36dbb\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " Nov 01 16:14:04.801159 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.801098 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca\") pod \"2b678595-2640-4558-acfa-2361f9f36dbb\" (UID: \"2b678595-2640-4558-acfa-2361f9f36dbb\") " Nov 01 16:14:04.801504 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.801470 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca" pod "2b678595-2640-4558-acfa-2361f9f36dbb" (UID: "2b678595-2640-4558-acfa-2361f9f36dbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:14:04.804206 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.804164 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca" pod "2b678595-2640-4558-acfa-2361f9f36dbb" (UID: "2b678595-2640-4558-acfa-2361f9f36dbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:14:04.804489 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.804466 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh" pod "2b678595-2640-4558-acfa-2361f9f36dbb" (UID: "2b678595-2640-4558-acfa-2361f9f36dbb"). InnerVolumeSpecName "kube-api-access-vfgjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:14:04.902753 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.902699 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:14:04.902753 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.902757 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b678595-2640-4558-acfa-2361f9f36dbb-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:14:04.903065 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:04.902776 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgjh\" (UniqueName: \"kubernetes.io/projected/2b678595-2640-4558-acfa-2361f9f36dbb-kube-api-access-vfgjh\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:14:05.523778 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:05.523734 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" event={"ID":"2b678595-2640-4558-acfa-2361f9f36dbb","Type":"ContainerDied","Data":"db0aff32918c5e6898df64f834bbe500a19dd0f2f0c8468b022ac3c73389b71c"} Nov 01 16:14:05.524016 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:05.523801 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db0aff32918c5e6898df64f834bbe500a19dd0f2f0c8468b022ac3c73389b71c" Nov 01 16:14:05.524016 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:05.523872 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fc48a7b-266ml" Nov 01 16:14:08.047334 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:08.047298 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp"] Nov 01 16:14:08.097151 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:08.097120 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e98ff841-7v9fp"] Nov 01 16:14:09.080383 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:09.080349 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65a5088-9995-43de-9295-62f1d36cf1b9" path="/var/lib/kubelet/pods/d65a5088-9995-43de-9295-62f1d36cf1b9/volumes" Nov 01 16:14:12.561305 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.561270 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c"] Nov 01 16:14:12.561809 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.561683 5473 cpu_manager.go:414] "containerMap: removing container" podUID="2b678595-2640-4558-acfa-2361f9f36dbb" containerName="revoke-keys" Nov 01 16:14:12.561809 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.561695 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b678595-2640-4558-acfa-2361f9f36dbb" containerName="revoke-keys" Nov 01 16:14:12.561809 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.561767 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b678595-2640-4558-acfa-2361f9f36dbb" containerName="revoke-keys" Nov 01 16:14:12.571630 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.571596 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.617049 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.617016 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c"] Nov 01 16:14:12.619042 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.619018 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.619205 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.619072 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.619205 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.619101 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdqt\" (UniqueName: \"kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.663914 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.663883 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:14:12.664071 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.663933 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:14:12.720497 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.720469 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.720659 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.720525 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.720659 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.720544 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdqt\" (UniqueName: \"kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.721122 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.721100 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.722781 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.722763 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.776909 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.776886 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdqt\" (UniqueName: \"kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt\") pod \"maas-api-revoke-sub-af5b0d9f-m792c\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:12.885050 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:12.884973 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:13.255930 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:13.255899 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c"] Nov 01 16:14:13.256623 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:14:13.256590 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839b3692_4e96_4ace_894d_4fa0612ddd15.slice/crio-6edfbbea513d865289e3b6a8a567cb821daafb28a1777d90297600dd3a53fd31 WatchSource:0}: Error finding container 6edfbbea513d865289e3b6a8a567cb821daafb28a1777d90297600dd3a53fd31: Status 404 returned error can't find the container with id 6edfbbea513d865289e3b6a8a567cb821daafb28a1777d90297600dd3a53fd31 Nov 01 16:14:13.599733 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:13.599687 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" event={"ID":"839b3692-4e96-4ace-894d-4fa0612ddd15","Type":"ContainerStarted","Data":"b573c963550441453853411d70a6f80a06cf03a5320b307d6d77cc11435fa2c1"} Nov 01 16:14:13.600336 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:13.599785 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" event={"ID":"839b3692-4e96-4ace-894d-4fa0612ddd15","Type":"ContainerStarted","Data":"6edfbbea513d865289e3b6a8a567cb821daafb28a1777d90297600dd3a53fd31"} Nov 01 16:14:13.714064 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:13.713046 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" podStartSLOduration=1.7130288870000001 podStartE2EDuration="1.713028887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:14:13.711913109 +0000 UTC m=+6992.770064960" watchObservedRunningTime="2026-11-01 16:14:13.713028887 +0000 UTC m=+6992.771180742" Nov 01 16:14:14.606771 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:14.606736 5473 generic.go:356] "Generic (PLEG): container finished" podID="839b3692-4e96-4ace-894d-4fa0612ddd15" containerID="b573c963550441453853411d70a6f80a06cf03a5320b307d6d77cc11435fa2c1" exitCode=0 Nov 01 16:14:14.607182 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:14.606811 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" event={"ID":"839b3692-4e96-4ace-894d-4fa0612ddd15","Type":"ContainerDied","Data":"b573c963550441453853411d70a6f80a06cf03a5320b307d6d77cc11435fa2c1"} Nov 01 16:14:16.254247 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.247065 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:16.343241 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.342734 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt\" (UniqueName: \"kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt\") pod \"839b3692-4e96-4ace-894d-4fa0612ddd15\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " Nov 01 16:14:16.349264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.343855 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca\") pod \"839b3692-4e96-4ace-894d-4fa0612ddd15\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " Nov 01 16:14:16.349264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.344503 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca" pod "839b3692-4e96-4ace-894d-4fa0612ddd15" (UID: "839b3692-4e96-4ace-894d-4fa0612ddd15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:14:16.349264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.344594 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca\") pod \"839b3692-4e96-4ace-894d-4fa0612ddd15\" (UID: \"839b3692-4e96-4ace-894d-4fa0612ddd15\") " Nov 01 16:14:16.349264 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.345391 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-openshift-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:14:16.358719 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.356353 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca" pod "839b3692-4e96-4ace-894d-4fa0612ddd15" (UID: "839b3692-4e96-4ace-894d-4fa0612ddd15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:14:16.374532 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.374463 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt" pod "839b3692-4e96-4ace-894d-4fa0612ddd15" (UID: "839b3692-4e96-4ace-894d-4fa0612ddd15"). InnerVolumeSpecName "kube-api-access-gpdqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:14:16.454991 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.454894 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/839b3692-4e96-4ace-894d-4fa0612ddd15-maas-api-xks-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:14:16.455249 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.455209 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdqt\" (UniqueName: \"kubernetes.io/projected/839b3692-4e96-4ace-894d-4fa0612ddd15-kube-api-access-gpdqt\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:14:16.676242 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.675872 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" Nov 01 16:14:16.680238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.677240 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af5b0d9f-m792c" event={"ID":"839b3692-4e96-4ace-894d-4fa0612ddd15","Type":"ContainerDied","Data":"6edfbbea513d865289e3b6a8a567cb821daafb28a1777d90297600dd3a53fd31"} Nov 01 16:14:16.680238 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:16.677298 5473 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6edfbbea513d865289e3b6a8a567cb821daafb28a1777d90297600dd3a53fd31" Nov 01 16:14:20.320463 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:20.320432 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:31.107197 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:14:31.107162 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:14:41.051541 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:41.051505 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl"] Nov 01 16:14:41.101052 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:41.101023 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl"] Nov 01 16:14:41.132359 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:14:41.132314 5473 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-fc9279f4-4lnbl\" is forbidden: User \"system:node:ip-10-0-131-109.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-131-109.us-west-2.compute.internal' and this object" podUID="8ef9c557-b2aa-48b8-9700-04a89aca9b31" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc9279f4-4lnbl" Nov 01 16:14:43.080414 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:43.080378 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef9c557-b2aa-48b8-9700-04a89aca9b31" path="/var/lib/kubelet/pods/8ef9c557-b2aa-48b8-9700-04a89aca9b31/volumes" Nov 01 16:14:46.161578 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:46.161544 5473 scope.go:122] "RemoveContainer" containerID="8d1b7609c28c6561f530b829ac5288eec0f8219eaec32672d8bca7cfdfb06877" Nov 01 16:14:46.170720 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:46.170696 5473 scope.go:122] "RemoveContainer" containerID="7bd575f943f422f2744013f9bd864817e328cee848c0c97e61c6701cb9985b58" Nov 01 16:14:53.705495 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:53.705418 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gwmh8_2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7/manager/0.log" Nov 01 16:14:55.828523 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:55.828494 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-02ce465c-tv82c_270fd8ca-10aa-4876-a14b-8772dc422587/revoke-keys/0.log" Nov 01 16:14:56.043772 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:56.043741 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1d74f66d-g52kx_2e84a7f2-5820-42c9-8adc-063afa5a3e0a/revoke-keys/0.log" Nov 01 16:14:56.254072 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:56.253991 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-206f88f6-qq7jj_1d02152c-6936-4a63-a7c5-312f761c81ef/revoke-keys/0.log" Nov 01 16:14:56.468048 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:56.468017 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2d87a446-9x6gd_06cd8cc4-796d-421c-92aa-81cadff0d454/revoke-keys/0.log" Nov 01 16:14:56.673370 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:56.673344 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3576bff8-92nvr_1c0ee04a-fd7b-4c62-9f24-eb41bb05faad/revoke-keys/0.log" Nov 01 16:14:56.892730 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:56.892700 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3bdbc843-7rx8h_98743823-17b3-41b7-8984-7bbfaf40f94a/revoke-keys/0.log" Nov 01 16:14:57.107010 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:57.106982 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ff0dc45-lslfj_4c89ea00-7bde-4818-986c-45ab75ee726a/revoke-keys/0.log" Nov 01 16:14:57.319919 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:57.319891 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-53509b25-z64nh_730900f8-2e88-46be-b3d1-a6cc250db6a8/revoke-keys/0.log" Nov 01 16:14:57.530922 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:57.530895 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7f8cd973-62g5w_73d435d7-79ab-4e9c-b1ff-555cfd6ba1f2/revoke-keys/0.log" Nov 01 16:14:57.756010 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:57.755979 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7fc48a7b-266ml_2b678595-2640-4558-acfa-2361f9f36dbb/revoke-keys/0.log" Nov 01 16:14:57.964230 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:57.964136 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-af5b0d9f-m792c_839b3692-4e96-4ace-894d-4fa0612ddd15/revoke-keys/0.log" Nov 01 16:14:58.176206 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:58.176174 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b12f4962-ps7fp_3d6a1fce-1ee4-4564-9922-daaed112fcfc/revoke-keys/0.log" Nov 01 16:14:58.396928 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:58.396903 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b5aa1b67-dj2cj_97aefbd7-a0da-4825-8613-1f5661149cff/revoke-keys/0.log" Nov 01 16:14:58.622147 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:58.622123 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d9072d42-6tlpl_89445aed-6806-4700-b764-cc3d4dc51275/revoke-keys/0.log" Nov 01 16:14:58.834157 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:58.834133 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ee0d5d16-6gddw_b2521c6c-10d1-4562-aff1-554407a77cca/revoke-keys/0.log" Nov 01 16:14:59.035830 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:59.035799 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f9ff808b-t6ggb_9c42abd1-977c-48b3-92d9-cab1b134de8b/revoke-keys/0.log" Nov 01 16:14:59.254372 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:14:59.254301 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-5rq29_a124ad3b-6c83-45d3-ac07-2ff5b7e690ba/postgres/0.log" Nov 01 16:15:00.164103 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.164072 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl"] Nov 01 16:15:00.164510 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.164490 5473 cpu_manager.go:414] "containerMap: removing container" podUID="839b3692-4e96-4ace-894d-4fa0612ddd15" containerName="revoke-keys" Nov 01 16:15:00.164510 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.164502 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839b3692-4e96-4ace-894d-4fa0612ddd15" containerName="revoke-keys" Nov 01 16:15:00.164600 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.164565 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="839b3692-4e96-4ace-894d-4fa0612ddd15" containerName="revoke-keys" Nov 01 16:15:00.169560 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.169544 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.220484 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.220448 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl"] Nov 01 16:15:00.257304 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.257280 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892495-bmdwl\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.257462 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.257352 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtsdw\" (UniqueName: \"kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw\") pod \"maas-api-key-cleanup-29892495-bmdwl\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.266487 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.266466 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-d6ds8\"" Nov 01 16:15:00.358650 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.358623 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtsdw\" (UniqueName: \"kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw\") pod \"maas-api-key-cleanup-29892495-bmdwl\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.358829 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.358714 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892495-bmdwl\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.359382 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.359363 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892495-bmdwl\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.415363 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.415300 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtsdw\" (UniqueName: \"kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw\") pod \"maas-api-key-cleanup-29892495-bmdwl\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.483341 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.483316 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:00.853882 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.853856 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl"] Nov 01 16:15:00.855001 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:15:00.854973 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5facefef_6f6b_4400_a4f2_d22e4b0933e1.slice/crio-a68847603bd17d24c72568af75057a2dd15c4afb73203829d8233ad422a25c97 WatchSource:0}: Error finding container a68847603bd17d24c72568af75057a2dd15c4afb73203829d8233ad422a25c97: Status 404 returned error can't find the container with id a68847603bd17d24c72568af75057a2dd15c4afb73203829d8233ad422a25c97 Nov 01 16:15:00.987963 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:00.987930 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" event={"ID":"5facefef-6f6b-4400-a4f2-d22e4b0933e1","Type":"ContainerStarted","Data":"a68847603bd17d24c72568af75057a2dd15c4afb73203829d8233ad422a25c97"} Nov 01 16:15:01.109951 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:15:01.109867 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:15:01.995048 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:01.994962 5473 generic.go:356] "Generic (PLEG): container finished" podID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerID="786938cbcd1315731ac1618102a1ec63a5b52e78afe7aa5fa1b66602ac58ef04" exitCode=60 Nov 01 16:15:01.995048 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:01.995018 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" event={"ID":"5facefef-6f6b-4400-a4f2-d22e4b0933e1","Type":"ContainerDied","Data":"786938cbcd1315731ac1618102a1ec63a5b52e78afe7aa5fa1b66602ac58ef04"} Nov 01 16:15:01.995573 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:01.995316 5473 scope.go:122] "RemoveContainer" containerID="786938cbcd1315731ac1618102a1ec63a5b52e78afe7aa5fa1b66602ac58ef04" Nov 01 16:15:03.003514 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:03.003484 5473 generic.go:356] "Generic (PLEG): container finished" podID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerID="fc2551d6d7b07c1d7399266909a1f4f7a8f7340bf7c6e2dcea71afd24187af7d" exitCode=60 Nov 01 16:15:03.003988 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:03.003574 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" event={"ID":"5facefef-6f6b-4400-a4f2-d22e4b0933e1","Type":"ContainerDied","Data":"fc2551d6d7b07c1d7399266909a1f4f7a8f7340bf7c6e2dcea71afd24187af7d"} Nov 01 16:15:03.003988 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:03.003649 5473 scope.go:122] "RemoveContainer" containerID="786938cbcd1315731ac1618102a1ec63a5b52e78afe7aa5fa1b66602ac58ef04" Nov 01 16:15:03.003988 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:03.003792 5473 scope.go:122] "RemoveContainer" containerID="fc2551d6d7b07c1d7399266909a1f4f7a8f7340bf7c6e2dcea71afd24187af7d" Nov 01 16:15:03.004150 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:15:03.004044 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892495-bmdwl_odh-ai-gateway-infra(5facefef-6f6b-4400-a4f2-d22e4b0933e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" Nov 01 16:15:03.057758 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:03.057736 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6gn9k_2a92488d-7096-4b4c-ab4f-79bf4dd16e94/manager/0.log" Nov 01 16:15:04.011508 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:04.011479 5473 scope.go:122] "RemoveContainer" containerID="fc2551d6d7b07c1d7399266909a1f4f7a8f7340bf7c6e2dcea71afd24187af7d" Nov 01 16:15:04.011907 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:15:04.011693 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892495-bmdwl_odh-ai-gateway-infra(5facefef-6f6b-4400-a4f2-d22e4b0933e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" Nov 01 16:15:04.725448 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:04.725414 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fc9ccddb-pkr2x_d95c16e8-7021-40dd-b862-49ac7798725f/kube-auth-proxy/0.log" Nov 01 16:15:05.476850 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:05.476818 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-56jt9_8a84ef62-7215-4e81-8585-2ed334e28a34/istio-proxy/0.log" Nov 01 16:15:06.314438 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:06.314406 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-l5vkl_aea2e96a-5411-4745-903f-275b69c10843/istio-proxy/0.log" Nov 01 16:15:06.637343 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:06.637272 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-p8scg_552f992f-89c3-43bf-8a2c-44665d612997/istio-proxy/0.log" Nov 01 16:15:07.740612 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:07.740584 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-x769v_bdad6cc9-742d-42e6-a82d-b8c8664ed362/istio-proxy/0.log" Nov 01 16:15:10.351304 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:10.351265 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k4fn8_fa4c309d-c000-4d8e-a84b-e5e132765d60/main/0.log" Nov 01 16:15:10.778947 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:10.778881 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-rw8z7_f2603428-d7ca-4679-9cf4-8cfdb207c5e9/main/0.log" Nov 01 16:15:11.261227 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:11.261197 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qcr5w_c6067073-4021-4e03-b1d7-03260b1618c9/main/0.log" Nov 01 16:15:11.787408 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:11.787379 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-tvrsz_8dcdd7f9-e49a-4a88-a4af-d888fc97551d/main/0.log" Nov 01 16:15:14.074918 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:14.074884 5473 scope.go:122] "RemoveContainer" containerID="fc2551d6d7b07c1d7399266909a1f4f7a8f7340bf7c6e2dcea71afd24187af7d" Nov 01 16:15:15.076758 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:15.076666 5473 generic.go:356] "Generic (PLEG): container finished" podID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerID="c76865f2b807299c9ec469584b87ae388776ed6ef93bbb87761f4fb309cc5ee7" exitCode=60 Nov 01 16:15:15.080443 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:15.080415 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" event={"ID":"5facefef-6f6b-4400-a4f2-d22e4b0933e1","Type":"ContainerDied","Data":"c76865f2b807299c9ec469584b87ae388776ed6ef93bbb87761f4fb309cc5ee7"} Nov 01 16:15:15.080574 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:15.080473 5473 scope.go:122] "RemoveContainer" containerID="fc2551d6d7b07c1d7399266909a1f4f7a8f7340bf7c6e2dcea71afd24187af7d" Nov 01 16:15:15.080815 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:15.080796 5473 scope.go:122] "RemoveContainer" containerID="c76865f2b807299c9ec469584b87ae388776ed6ef93bbb87761f4fb309cc5ee7" Nov 01 16:15:15.081079 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:15:15.081057 5473 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892495-bmdwl_odh-ai-gateway-infra(5facefef-6f6b-4400-a4f2-d22e4b0933e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" Nov 01 16:15:16.204445 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.204408 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl"] Nov 01 16:15:16.340661 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.340636 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:16.406234 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.406172 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca\") pod \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " Nov 01 16:15:16.406399 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.406350 5473 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw\" (UniqueName: \"kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw\") pod \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\" (UID: \"5facefef-6f6b-4400-a4f2-d22e4b0933e1\") " Nov 01 16:15:16.406600 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.406578 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca" pod "5facefef-6f6b-4400-a4f2-d22e4b0933e1" (UID: "5facefef-6f6b-4400-a4f2-d22e4b0933e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:16.406681 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.406667 5473 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5facefef-6f6b-4400-a4f2-d22e4b0933e1-maas-api-service-ca\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:16.408380 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.408352 5473 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw" pod "5facefef-6f6b-4400-a4f2-d22e4b0933e1" (UID: "5facefef-6f6b-4400-a4f2-d22e4b0933e1"). InnerVolumeSpecName "kube-api-access-rtsdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:16.508111 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:16.508025 5473 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtsdw\" (UniqueName: \"kubernetes.io/projected/5facefef-6f6b-4400-a4f2-d22e4b0933e1-kube-api-access-rtsdw\") on node \"ip-10-0-131-109.us-west-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:17.091705 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:17.091682 5473 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" Nov 01 16:15:17.091837 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:17.091711 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl" event={"ID":"5facefef-6f6b-4400-a4f2-d22e4b0933e1","Type":"ContainerDied","Data":"a68847603bd17d24c72568af75057a2dd15c4afb73203829d8233ad422a25c97"} Nov 01 16:15:17.091837 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:17.091771 5473 scope.go:122] "RemoveContainer" containerID="c76865f2b807299c9ec469584b87ae388776ed6ef93bbb87761f4fb309cc5ee7" Nov 01 16:15:17.255563 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:17.255529 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl"] Nov 01 16:15:17.305384 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:17.305349 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892495-bmdwl"] Nov 01 16:15:19.080468 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:19.080433 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" path="/var/lib/kubelet/pods/5facefef-6f6b-4400-a4f2-d22e4b0933e1/volumes" Nov 01 16:15:20.596539 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:20.596506 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:21.121239 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121182 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47rht/must-gather-jmv5x"] Nov 01 16:15:21.121589 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121575 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121589 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121590 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121710 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121606 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121710 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121611 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121710 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121680 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121710 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121689 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121864 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121761 5473 cpu_manager.go:414] "containerMap: removing container" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121864 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121767 5473 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.121864 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.121823 5473 memory_manager.go:365] "RemoveStaleState removing state" podUID="5facefef-6f6b-4400-a4f2-d22e4b0933e1" containerName="cleanup" Nov 01 16:15:21.126470 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.126450 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.156082 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.156056 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:21.231778 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.231747 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47rht\"/\"openshift-service-ca.crt\"" Nov 01 16:15:21.231778 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.231777 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47rht\"/\"kube-root-ca.crt\"" Nov 01 16:15:21.232042 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.231923 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47rht\"/\"default-dockercfg-nzz7m\"" Nov 01 16:15:21.236020 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.235988 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47rht/must-gather-jmv5x"] Nov 01 16:15:21.251300 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.251278 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a37be346-56b6-4733-a1d9-80d5634798d8-must-gather-output\") pod \"must-gather-jmv5x\" (UID: \"a37be346-56b6-4733-a1d9-80d5634798d8\") " pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.251410 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.251308 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtvt\" (UniqueName: \"kubernetes.io/projected/a37be346-56b6-4733-a1d9-80d5634798d8-kube-api-access-vmtvt\") pod \"must-gather-jmv5x\" (UID: \"a37be346-56b6-4733-a1d9-80d5634798d8\") " pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.329369 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.329339 5473 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:21.352567 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.352542 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a37be346-56b6-4733-a1d9-80d5634798d8-must-gather-output\") pod \"must-gather-jmv5x\" (UID: \"a37be346-56b6-4733-a1d9-80d5634798d8\") " pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.352713 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.352577 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtvt\" (UniqueName: \"kubernetes.io/projected/a37be346-56b6-4733-a1d9-80d5634798d8-kube-api-access-vmtvt\") pod \"must-gather-jmv5x\" (UID: \"a37be346-56b6-4733-a1d9-80d5634798d8\") " pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.352902 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.352883 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a37be346-56b6-4733-a1d9-80d5634798d8-must-gather-output\") pod \"must-gather-jmv5x\" (UID: \"a37be346-56b6-4733-a1d9-80d5634798d8\") " pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.412041 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.411961 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtvt\" (UniqueName: \"kubernetes.io/projected/a37be346-56b6-4733-a1d9-80d5634798d8-kube-api-access-vmtvt\") pod \"must-gather-jmv5x\" (UID: \"a37be346-56b6-4733-a1d9-80d5634798d8\") " pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.440507 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.440478 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47rht/must-gather-jmv5x" Nov 01 16:15:21.817773 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.817742 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47rht/must-gather-jmv5x"] Nov 01 16:15:21.818292 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:15:21.818076 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda37be346_56b6_4733_a1d9_80d5634798d8.slice/crio-2ff584a33e9b25587c4d82b77ef5347e72e8ae2a3a57cc02b6e5e95f367952d8 WatchSource:0}: Error finding container 2ff584a33e9b25587c4d82b77ef5347e72e8ae2a3a57cc02b6e5e95f367952d8: Status 404 returned error can't find the container with id 2ff584a33e9b25587c4d82b77ef5347e72e8ae2a3a57cc02b6e5e95f367952d8 Nov 01 16:15:21.820077 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:21.820059 5473 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:15:22.124006 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:22.123921 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47rht/must-gather-jmv5x" event={"ID":"a37be346-56b6-4733-a1d9-80d5634798d8","Type":"ContainerStarted","Data":"2ff584a33e9b25587c4d82b77ef5347e72e8ae2a3a57cc02b6e5e95f367952d8"} Nov 01 16:15:23.046744 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:23.046711 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb"] Nov 01 16:15:23.096831 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:23.096805 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9ff808b-t6ggb"] Nov 01 16:15:25.080310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:25.080278 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c42abd1-977c-48b3-92d9-cab1b134de8b" path="/var/lib/kubelet/pods/9c42abd1-977c-48b3-92d9-cab1b134de8b/volumes" Nov 01 16:15:30.304576 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:30.304527 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ckfsz_99114e06-1d8b-4e06-aa86-4436de92e794/global-pull-secret-syncer/0.log" Nov 01 16:15:31.113123 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:15:31.113092 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:15:31.197625 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:31.197582 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47rht/must-gather-jmv5x" event={"ID":"a37be346-56b6-4733-a1d9-80d5634798d8","Type":"ContainerStarted","Data":"c7c4b04a3f7450cabc5ecb7ac359606ff445faed915277353b3631ab86bb00b2"} Nov 01 16:15:32.205716 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:32.205678 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47rht/must-gather-jmv5x" event={"ID":"a37be346-56b6-4733-a1d9-80d5634798d8","Type":"ContainerStarted","Data":"577b753e5312de103bcbada35d8461684807493423dc5833e9f603fc9e3624cc"} Nov 01 16:15:32.319595 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:32.319539 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47rht/must-gather-jmv5x" podStartSLOduration=2.093455966 podStartE2EDuration="11.319525003s" totalImagesPullingTime="9.226069037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:32.318257719 +0000 UTC m=+7071.376409568" watchObservedRunningTime="2026-11-01 16:15:32.319525003 +0000 UTC m=+7071.377676879" Nov 01 16:15:33.399156 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:33.399103 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-82phh_b241b884-12de-43ee-af05-7a92bff07b42/konnectivity-agent/0.log" Nov 01 16:15:34.046521 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:34.046484 5473 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c"] Nov 01 16:15:34.096384 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:34.096349 5473 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02ce465c-tv82c"] Nov 01 16:15:34.880747 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:34.880720 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-109.us-west-2.compute.internal_fbed3511ed9fe6dd871e46aab86e24ee/haproxy/0.log" Nov 01 16:15:35.080094 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:35.080054 5473 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270fd8ca-10aa-4876-a14b-8772dc422587" path="/var/lib/kubelet/pods/270fd8ca-10aa-4876-a14b-8772dc422587/volumes" Nov 01 16:15:38.162668 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:38.162616 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gwmh8_2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7/manager/0.log" Nov 01 16:15:40.978279 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:40.978249 5473 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s"] Nov 01 16:15:40.983596 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:40.983580 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.035669 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.035649 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"kube-root-ca.crt\"" Nov 01 16:15:41.086535 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.086493 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"openshift-service-ca.crt\"" Nov 01 16:15:41.086770 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.086570 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8k5v4\"/\"default-dockercfg-tf4nc\"" Nov 01 16:15:41.089715 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.089688 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s"] Nov 01 16:15:41.154738 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.154705 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-lib-modules\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.154896 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.154762 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-proc\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.154896 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.154866 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-podres\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.155014 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.154975 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-sys\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.155087 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.155068 5473 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hcmf\" (UniqueName: \"kubernetes.io/projected/010b058d-792a-451f-97fa-e5dc9fb01a1a-kube-api-access-7hcmf\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256038 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.255949 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-sys\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256038 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256001 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hcmf\" (UniqueName: \"kubernetes.io/projected/010b058d-792a-451f-97fa-e5dc9fb01a1a-kube-api-access-7hcmf\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256042 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-lib-modules\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256072 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-proc\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256075 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-sys\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256096 5473 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-podres\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256146 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-podres\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256186 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-lib-modules\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.256310 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.256177 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/010b058d-792a-451f-97fa-e5dc9fb01a1a-proc\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.361465 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.361439 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"kube-root-ca.crt\"" Nov 01 16:15:41.421962 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.421930 5473 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8k5v4\"/\"openshift-service-ca.crt\"" Nov 01 16:15:41.433170 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.433150 5473 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hcmf\" (UniqueName: \"kubernetes.io/projected/010b058d-792a-451f-97fa-e5dc9fb01a1a-kube-api-access-7hcmf\") pod \"perf-node-gather-daemonset-ljm2s\" (UID: \"010b058d-792a-451f-97fa-e5dc9fb01a1a\") " pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:41.648356 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.648323 5473 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8k5v4\"/\"default-dockercfg-tf4nc\"" Nov 01 16:15:41.657420 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:41.657384 5473 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:42.036570 ip-10-0-131-109 kubenswrapper[5473]: W1101 16:15:42.036528 5473 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod010b058d_792a_451f_97fa_e5dc9fb01a1a.slice/crio-b057815ac862e3e935c3d93cb4033c6ddeef6178fa24ea9c8a233d0f5b3adc29 WatchSource:0}: Error finding container b057815ac862e3e935c3d93cb4033c6ddeef6178fa24ea9c8a233d0f5b3adc29: Status 404 returned error can't find the container with id b057815ac862e3e935c3d93cb4033c6ddeef6178fa24ea9c8a233d0f5b3adc29 Nov 01 16:15:42.037849 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:42.037823 5473 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s"] Nov 01 16:15:42.270381 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:42.270341 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" event={"ID":"010b058d-792a-451f-97fa-e5dc9fb01a1a","Type":"ContainerStarted","Data":"0b970096a26e34fed24cdebe29bfbe3f0423758440d6d41c71eb02e494a832e0"} Nov 01 16:15:42.270558 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:42.270408 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:42.270558 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:42.270434 5473 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" event={"ID":"010b058d-792a-451f-97fa-e5dc9fb01a1a","Type":"ContainerStarted","Data":"b057815ac862e3e935c3d93cb4033c6ddeef6178fa24ea9c8a233d0f5b3adc29"} Nov 01 16:15:42.385372 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:42.385258 5473 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" podStartSLOduration=2.385242858 podStartE2EDuration="2.385242858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 16:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:42.382814969 +0000 UTC m=+7081.440966820" watchObservedRunningTime="2026-11-01 16:15:42.385242858 +0000 UTC m=+7081.443394701" Nov 01 16:15:43.542179 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:43.542113 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-6gn9k_2a92488d-7096-4b4c-ab4f-79bf4dd16e94/manager/0.log" Nov 01 16:15:44.547773 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:44.547741 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gwmh8_2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7/manager/0.log" Nov 01 16:15:46.049999 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:46.049964 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/alertmanager/0.log" Nov 01 16:15:46.198706 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:46.198660 5473 scope.go:122] "RemoveContainer" containerID="bb1d77b1211e48720c87db0bfc9da0925c765ed8885652d471a217a98c3c8ca3" Nov 01 16:15:46.208016 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:46.207989 5473 scope.go:122] "RemoveContainer" containerID="7f1644a2b25b14ce166c790118fc8ba39482b5d47779b7d7e205d711921f40b0" Nov 01 16:15:46.216793 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:46.216771 5473 scope.go:122] "RemoveContainer" containerID="71cf9004be1aa112c3b3236707b7621ea6752c5817b573453f3f2b3a10e99df6" Nov 01 16:15:46.471421 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:46.471338 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/config-reloader/0.log" Nov 01 16:15:46.892667 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:46.892632 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/kube-rbac-proxy-web/0.log" Nov 01 16:15:47.313638 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:47.313609 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/kube-rbac-proxy/0.log" Nov 01 16:15:47.735195 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:47.735118 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/kube-rbac-proxy-metric/0.log" Nov 01 16:15:48.156423 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:48.156386 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/prom-label-proxy/0.log" Nov 01 16:15:48.288681 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:48.288655 5473 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8k5v4/perf-node-gather-daemonset-ljm2s" Nov 01 16:15:48.578072 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:48.578040 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_496d04ed-d532-4793-a407-e0cea3c23e2a/init-config-reloader/0.log" Nov 01 16:15:50.762017 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:50.761985 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-674fc8bc5b-hm9rs_b6a34a13-3327-4384-bc2d-09e30659af46/metrics-server/0.log" Nov 01 16:15:51.565001 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:51.564961 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gwmh8_2ef0fc4f-c327-4c76-98ae-b1d7bd97d8b7/manager/0.log" Nov 01 16:15:52.837515 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:52.837481 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pg8hh_03b6d85d-bd06-4e6c-9b9b-93fc59812d24/node-exporter/0.log" Nov 01 16:15:53.257703 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:53.257634 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pg8hh_03b6d85d-bd06-4e6c-9b9b-93fc59812d24/kube-rbac-proxy/0.log" Nov 01 16:15:53.677134 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:53.677107 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pg8hh_03b6d85d-bd06-4e6c-9b9b-93fc59812d24/init-textfile/0.log" Nov 01 16:15:56.642245 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:56.642194 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/0.log" Nov 01 16:15:56.643422 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:56.643402 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/prometheus/1.log" Nov 01 16:15:56.960199 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:56.960118 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/config-reloader/0.log" Nov 01 16:15:57.383133 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:57.383096 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/thanos-sidecar/0.log" Nov 01 16:15:57.713614 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:57.713537 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fc9ccddb-pkr2x_d95c16e8-7021-40dd-b862-49ac7798725f/kube-auth-proxy/0.log" Nov 01 16:15:57.804124 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:57.804095 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/kube-rbac-proxy-web/0.log" Nov 01 16:15:58.224360 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:58.224332 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/kube-rbac-proxy/0.log" Nov 01 16:15:58.354536 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:58.354508 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-56jt9_8a84ef62-7215-4e81-8585-2ed334e28a34/istio-proxy/0.log" Nov 01 16:15:58.644717 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:58.644689 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/kube-rbac-proxy-thanos/0.log" Nov 01 16:15:59.012901 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:59.012875 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-l5vkl_aea2e96a-5411-4745-903f-275b69c10843/istio-proxy/0.log" Nov 01 16:15:59.068162 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:59.068135 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1a82d731-8a69-4a50-8246-8e3ac6363720/init-config-reloader/0.log" Nov 01 16:15:59.232496 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:59.232468 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-p8scg_552f992f-89c3-43bf-8a2c-44665d612997/istio-proxy/0.log" Nov 01 16:15:59.497983 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:59.497902 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-j9hdh_61db8c12-1910-483d-9ad4-8c491354c549/prometheus-operator/0.log" Nov 01 16:15:59.913033 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:15:59.913002 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-j9hdh_61db8c12-1910-483d-9ad4-8c491354c549/kube-rbac-proxy/0.log" Nov 01 16:16:00.093913 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:16:00.093881 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-x769v_bdad6cc9-742d-42e6-a82d-b8c8664ed362/istio-proxy/0.log" Nov 01 16:16:00.744063 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:16:00.744034 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c9bbb4584-9lxxr_aff6a7b6-cf63-43c0-ba97-06900306308d/telemeter-client/0.log" Nov 01 16:16:01.120434 ip-10-0-131-109 kubenswrapper[5473]: E1101 16:16:01.120394 5473 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4279c" containerName="extractor" Nov 01 16:16:01.165408 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:16:01.165379 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c9bbb4584-9lxxr_aff6a7b6-cf63-43c0-ba97-06900306308d/reload/0.log" Nov 01 16:16:01.587674 ip-10-0-131-109 kubenswrapper[5473]: I1101 16:16:01.587648 5473 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c9bbb4584-9lxxr_aff6a7b6-cf63-43c0-ba97-06900306308d/kube-rbac-proxy/0.log"